Hacker News Reader

Comment by admax88q | Hacker News Reader

There's a lot of lament that this bypasses the operating system level config, but I feel like we wouldn't be here if the operating system had stepped up their DNS any time in the last several years.

This is a workaround for the fact that many networks are hostile to DNS lookups (injecting ads, tracking, captive portals), and operating systems continue to just default to the ISPs DNS provider, which is often hostile.

OS's could have switched to running their own local caching resolver ala unbound years ago, could have integrated DNSCrypt or DoT years ago. But instead the defaults are still horribly unsecure for the average user.

For the techie user who's running PiHole or some custom local DNS, yeah this will be annoying, but for the average user who's just delegating to their ISP that is then tracking their DNS lookups and injecting ads if they mistype a domain, this is a win.

View on HN

So the fact that some ISPs are apparently untrustworthy is enough to blow up the whole thing which has worked pretty well for the past 30 years?

There are plenty of valid use cases for captive portals. But yes, injecting ads and tracking are obviously messed up, but DNS is only a tiny part of that problem, and I don't see how using DoH will help with them. If you want to get rid those, then get rid of shitty ISPs. This is throwing out the baby with the bathwater.

This will simply lead to ISPs and network operators blocking DoH. And instead of a wonderfully decentralized DNS, we'll end up with centralized DoH in the hands of Cloudflare and their ilk. Who have been shown repeatedly to be user hostile; especially towards privacy conscious users.

Can they block DoH, with reasonable tradeoffs? How?

And will they care enough? I get the sense that they don't really gain that much from messing with DNS, relatively.

They couldn't necessarily block all DoH, but they could destroy its prevalence by just blocking all of the default resolvers and all of the resolvers that the browsers suggest to users. Those are public lists.

I suspect you're right that they don't get that much value from messing with DNS, though... not enough to deal with the user complaints they'd get. So they probably wouldn't bother unless pressured.

However, they might decide to "play chicken" with Mozilla and other sources of DoH apps by seeing who yielded to the user complaints first. There'd be a big temptation for Mozilla to fall back to "regular" DNS if DoH failed, and then the ISP would effectively win.

So the fact that some ISPs are apparently untrustworthy is enough to blow up the whole thing which has worked pretty well for the past 30 years?

Right. In place of that, we get one central point of DNS censorship. Where any unapproved site can be made to disappear.

"CIRA Canadian Shield" "Truly free". Right. "Freedom is Slavery" is more like it. In the US, Mozilla is apparently putting users under the thumb of Cloudflare.[1]

I'd rather have a mode where regular DNS and centrally controlled DNS are compared, and you get a popup if they differ. That warns you of censorship attempts. But that doesn't seem to be an option.

[1] https://forum.netgate.com/topic/133679/heads-up-be-aware-of-...

> In the US, Mozilla is apparently putting users under the thumb of Cloudflare.

Even if you don't think Cloudflare is perfect, don't you trust them way more than, e.g., Comcast?

> I'd rather have a mode where regular DNS and centrally controlled DNS are compared, and you get a popup if they differ. That warns you of censorship attempts. But that doesn't seem to be an option.

This would only fix the censorship problem, not the surveillance problem.

> Even if you don't think Cloudflare is perfect, don't you trust them way more than, e.g., Comcast?

Why would I trust a company I have no binding contract with, that doesn’t gain anything from my using their services, that doesn’t stand to lose anything if they lose me and others as a user of this particular service, etc over a company I entered into a legal agreement with, pay a monthly sum to provide a named service, and can sue in a court of law with some modicum of standing if things go sour?

You can't sue Comcast; you signed away your right to do so, individually and as a class, when you signed up for their service[1]. Most ISPs have class action waivers and binding arbitration, these days. In fact, you explicitly agreed to let them aggregate your data and sell it to others for their profit[2], assist third-parties in targeting you for advertisements[3], and combine data they have on file with you from other companies databases[4].

So, no, if I were in your shoes, I wouldn't trust Comcast at all.

[1]: Comcast Agreement for Residential Service, Section 13 and 14. https://www.xfinity.com/Corporate/Customers/Policies/Subscri...

[2]: Xfinity Privacy Policy. https://www.xfinity.com/privacy/policy "Create measurement and analytics reports for us and others"

[3]: Ibid. "We use the information we collect to... deliver personalized marketing and advertising for our own and others’ products and services"

[4]: Ibid. "We may combine information across our systems, platforms, and databases. This includes combining information we receive from third parties and information about your use of our Services."

> you signed away your right to do so, individually and as a class

Has this ever been tested in court? Specifically in a situation of a service almost everyone needs and they don't have a choice in the company? The contract can say lots of things, but none of them have to be legally binding.

Because Comcast has a repeated, long-term pattern of manipulating the contract to be one-sided against you, of taking every bad action that the contract might even possibly allow, and arguably of violating the contract, whereas Cloudflare does not have a comparable pattern of such actions?

Because there's about a 99 percent chance that the contract is either completely silent about the matters of immediate concern, or has some language that implies they are actively allowed to act badly in those ways; the contract therefore offers you absolutely zero protection; and there is absolutely no way you can renegotiate the contract?

Lawsuits are incredibly impractical, and that's assuming you even ever detect the behavior you expect to be suing over.

> In place of that, we get one central point of DNS censorship.

Just don't use Cloudflare as your DoH resolver. What is preventing DNS from becoming centralized, and why is DoH different? You can still use any resolver you want, and network-wide settings will come out once network hardware and operating systems catch up to the ecosystem and start implementing their own support.

It is easier to block DNS queries and censor specific sites over unencrypted DNS than it is to block specific sites when you can encrypt your queries to any DoH provider, even ones that are running locally on your own network.

In fact, one of the primary criticisms of DoH from UK ISPs is that it will make it harder for them to block pirate sites and fulfill court orders to block websites[0]. When you listen to the actual censors, they all want DNS to stick around. That should tell you something about whether DoH really is a tool for censorship.

> I'd rather have a mode where regular DNS and centrally controlled DNS are compared

Why would this be harder to do with DoH than it would be to do with DNS?

[0]: https://indico.uknof.org.uk/event/46/contributions/668/attac...

> I'd rather have a mode where regular DNS and centrally controlled DNS are compared, and you get a popup if they differ. That warns you of censorship attempts. But that doesn't seem to be an option.

This is difficult with CDNs because it's commonly intended that you get a different RR-set response depending on where your query originated from. So, somewhat awkwardly, there's no unique correct answer to a particular DNS query in many cases today.

I just looked up IN A www.google.com from two different networks, and got two different answers, yet I think both of them are correct and are the answers that Google intended for me to get!

(These discrepancies are themselves one reason that CDNs have been wary of large numbers of users not using ISP-provided recursive resolvers. There is a sort of compromise workaround called ECS, https://en.wikipedia.org/wiki/EDNS_Client_Subnet, in which the recursive resolver gives the authoritative server a hint about where the query came from, to facilitate giving a response that may be most relevant to that specific client, typically a nearby CDN instance.)

A useful check is to then compare the certificates presented by the sites the DNS wants you to talk to. If both paths present the same cert, you're good. If they present different certs, there's a problem.

Cloudflare's creation of fake TLS certs for sites they front-end may mess that up, though. Do you see the same TLS cert from all Cloudflare MITM nodes?

Would a networking expert with some historical perspective then not say that we're misusing DNS for routing or load balancing?

I'm sure there was a huge debate about this when CDNs first started doing it. (I think Akamai was already doing it 20 years ago or so, so it wasn't just invented now.)

You could, of course, try to handle it at a higher protocol layer. For example, you could try to dynamically generate HTML like

for one visitor, whereas a different visitor would get

This would mean that latency for an initial HTTPS page load would be higher, but then for subresources and maybe other pages on the site it would be lower. But there are other difficulties, like requiring the web server to generate page content (at least slightly) dynamically instead of statically.

I'm sure that there are many people who felt that there was something seriously improper about intentionally serving different RRs within the same zone at the same point in time. From what I've heard, this practice is so common that most people working on related Internet standards just assume that this practice has to be accommodated because CDNs and their customers will definitely refuse to give it up. (I know this issue came up a ton in ESNI because in the first version there was a risk that you could cache one version of an A record and another version of an ESNI key record, so the two could be out of sync and therefore inconsistent, if they pointed to different CDNs or different CDN instances that had different public keys. A "nice" solution might be to say "well, don't create deliberately inconsistent sets of DNS records in your DNS zone, then" but apparently nobody found that solution at all acceptable for performance, so they created a more complicated workaround where both pieces of information are merged into a single rrtype.)

"I'd rather have a mode where regular DNS and centrally controlled DNS are compared, and you get a popup if they differ."

Not sure about "mode" and "popup" as I am not a fan of browsers and GUIs but I have been comparing results from about 50 or so DoH providers. I use DoH without a browser to gather DNS data in bulk.

We all have a clear idea about what "censorship" means but as we go further down the rabbit hole of myriad third party, non-ISP DNS providers how do we distinguish between "filtering" and "censorship". Projects like OpenDNS made a business out of DNS filtering. Now we are seeing others trying to make a business out of filtering ads.

It is well-known from studies using different DNS vantage points (looking glass) that DNS-based censorship is active in some countries and also that the number of countries doing this is generally rising. On a personal level, I do see differences in results from different DoH providers. What really bothered me recently was one provider returning "0.0.0.0" for certain domains, I am guessing because they were websites that report on the online ad industry. I am OK if they want to exclude certain domains, but returning the network equivalent of "any address" is, IMO, not an acceptable way to block access.

Personally, I think relying wholly on third party DNS is a bad idea. It is a useful resource, it's good to have many sources of DNS data, but third party operated DNS caches will never be authoritative for any domain. These third parties are usually just more internet middleman who want to make money from being in the middle of users network traffic. We always need access to the authoritative servers for each domain. That's our "source of truth".

DoH can be useful because I can download DNS data in bulk over TCP using HTTP pipelining. I load bulk data into haproxy's memory via a map file and I never need to do a single remote lookup. I run an authoritative DNS server that just answers all queries with the address of the proxy. Even more, I can add or delete entries from the map without restarting the proxy. The map is just a simple two column text file. This makes monitoring changes in DNS easy.

While I do not need "modes" and "popups" or other GUI-style conveniences, I can build a "no DNS" system into an OS that boots from USB stick into memory, never touches "disk" (can pull out the USB stick) and runs on a wide variety of hardware.

The "some ISPs" is really important here.

This is very much a US thing, I don't know as much about the situation in Canada, might be as bad. I really hope that this is not the begging of a roll out to the rest of the world.

Yeah agree, ISP choice here in Australia (at least in the metro areas) is plentiful and in my experience all my ISPs have behaved pretty ethically. I'm on Aussie Broadband now and their service has been fantastic, they even let you run your own line tests and connection resets from their customer portal.

I spent a number of years in the US at the mercy of Comcast and Later TWC in new york city and it was a very different experience. Also had a weird experience in Chicago where my building provided internet to all the Apartments with ads injected on 404 pages and the like. Really odd and no option to change.

All ISP dns servers in Australia are malicious. By law they must log all of your requests tied to your ID and they must block copyright infringement sites.

The first one alone is reason to have DoH enabled by default for all users in Australia.

When I lived there a few years ago my ISP was frustratingly adversarial.

https://news.ycombinator.com/item?id=6993831

> This is very much a US thing

DNS censorship is absolutely a thing in Germany and the UK for example. While here in Germany we might not get injected ads and other stuff, there are still more than enough problems to not make this a "US thing"

> If you want to get rid those, then get rid of shitty ISPs.

Good luck, when 30M+ only have DSL as an option, and >70% only have one Cable provider, and one DSL provider.

FCC Data: https://broadbandmap.fcc.gov/#/area-comparison?version=jun20...

More Info: https://www.theverge.com/22177154/us-internet-speed-maps-com...

> This will simply lead to ISPs and network operators blocking DoH.

With eSNI/ECH, browsers would just have to hardcode a bootstrap IP, and then ISPs won't be able to block DoH without also blocking all of every major CDN.

yup, will have to block eSNI (disable all hosts on firewall unless looked up on local DNS, per application)

Maybe this will bring back a form of domain fronting. Let the user put in a list of hosts that your tyrannical dictatorship allows through its firewall, and then do a fake DNS lookup through your local DNS server for them, then use the resulting IP to access other hosts on the same CDN.

Might still work. All DoH requests go to the same IP from the "major" proviers.

> which has worked pretty well for the past 30 years?

But DNS hasn't worked pretty well for the past 30 years. It's been abused for a pretty long time now, both for hijacking requests and for violating privacy.

I don't think that DoH advocates looked at a perfectly good system and just decided to replace it. They looked at a system with significant flaws, saw that this system was holding back consumer privacy and security, and proposed an alternative.

> but DNS is only a tiny part of that problem [...] If you want to get rid those, then get rid of shitty ISPs. This is throwing out the baby with the bathwater.

DNS is a huge part of that problem, because without encrypted DNS most people can't realistically keep their DNS queries private. What we're seeing is that we can do all kinds of fancy stuff around encrypting connections, but the DNS is always going to be a hole, and it's always going to make the other protections much less useful.

We want to eventually plug other holes that make websites identifiable. During that process, there are going to be periods where some of the holes are plugged, and some aren't. But that doesn't mean all of the efforts are useless, it means we're in the process of plugging security holes. The fact that we haven't widely deployed SNI encryption or TLS 1.3 does not mean that encrypted DNS is useless, it means that multiple technologies are required to work together to make progress towards meaningfully masking domain names.

We still have very hard problems to solve, like IP addresses themselves and statistical analysis of response sizes to determine which pages are being visited. But it's not much use trying to mitigate those problems if we can't even secure DNS resolution.

In regards to ISPs, in general (where possible) we like to avoid the model of "everyone is insecure but there are no bad actors." Imagine if we had rejected HTTPS because "this is only really a problem if your router is compromised or the ISPs are snooping." We have a solution now that helps reduce the amount of trust we need for ISPs. That's good, we should deploy it. Even if ISPs weren't a problem, it would be worth deploying.

> This will simply lead to ISPs and network operators blocking DoH. And instead of a wonderfully decentralized DNS, we'll end up with centralized DoH in the hands of Cloudflare and their ilk.

I don't see a lot of reason to believe this. First, there was nothing preventing ISPs from blocking DNS and forcing you to use their DNS resolvers in the past, so I don't see what DoH changes about that situation -- other than that it's a meaningful privacy increase, which might make bad actors more aggressive about blocking it. But surely we're not claiming that we should abandon meaningful privacy improvements because we're scared of what ISPs might do.

Second, DoH is decentralized, you can still use other resolvers other than Cloudflare, and you can still chain resolvers together. For the moment, I happen to recommend Cloudflare because I think it's offering more believable privacy promises than other providers, but I understand the concerns about Cloudflare's centralization. It's valid to use something else.

But seriously, what is centralized about DoH that isn't centralized about DNS? The fact that requests can't be hijacked? Did the Internet become centralized specifically because you can't MITM HTTPS requests now? I'm seeing people claim that DoH means every request will go to Cloudflare, and that's just flat out not true: anyone can set up a DoH resolver, even locally on their own network.

Okay, granted, not everyone on your network might respect your DoH settings you put on your Piihole or router, but that's not really different from today. Google already doesn't respect network settings for its DNS resolution on Chromecasts (or at least didn't last I checked). My browser already allows me to override my network settings for DNS. DoH changes nothing about that other than that OSes and routers are lagging on implementing network-wide DoH settings. If you think that staying with DNS would mean that IOT devices will perpetually respect your network settings for domain resolution, then I have bad news for you, because they're definitely going to stop doing that in the future regardless of what technology exists, and some of them have already stopped doing that today.

> Second, DoH is decentralized, you can still use other resolvers other than Cloudflare, and you can still chain resolvers together.

... but in practice almost all Firefox users will end up using one of the very few designated Mozilla TRRs. That's an increase in centralization, as measured on the ground. Social and behavioral effects count.

You can't claim one minute that "we're doing this to take care of naive users who can't protect themselves", and the next minute that "users can set up any arcane configuration they want".

In the end, you screw the naive users by increasing their practical centralization, and you screw the sophisticated users by adding complexity, breaking whatever protections they already have in place, and making it impossible for them to secure all of the applications on their systems at the same time.

Oh, and on edit, I guess I should address "first" as well as "second":

Nothing was stopping ISPs from blocking alternative DNS before... but very few people were using alternative DNS before, so it didn't have a lot of impact on ISPs' priorities. If you suddenly take away the ISPs' visibility into many people's DNS, then the ISPs get a reason to care. All kinds of things skate under the radar if they're not deployed at scale, but provoke responses when they get big enough to be threatening.

> ... but in practice almost all Firefox users will end up using one of the very few designated Mozilla TRRs. That's an increase in centralization, as measured on the ground. Social and behavioral effects count.

If we're going to go down this road and talk about defaults as they exist on the ground, then the reality is that most nontechnical users should be using Cloudflare's DoH resolver at the moment. Are there concerns about centralization? Yeah, of course. I'm as worried as anyone else about Cloudflare controlling the Internet, they are fundamentally a dangerous company. Being big, on its own, makes Cloudflare untrustworthy. But I'm also looking at Cloudflare's actions in the real world today, and they still have a better privacy record and are currently more trustworthy than any ISP in the US. Choosing them as a default in the US was the correct choice for Mozilla to make if you care about people's actual privacy in the real world today.

Mozilla's defaults here are also not set in stone. As network operators catch up with the technologies they've been ignoring, we'll get better OS and network integration for DoH settings. And as more providers start offering DoH, the defaults can change. Cloudflare is the default right now not because of a conspiracy but because regardless of how nervous they make us, and regardless of whether it's a good idea in theory in the future to have so much traffic routed through them, currently on the ground right now they are the best choice for most nontechnical users.

Additionally, currently on the ground there are few companies that are doing more than Cloudflare to decentralize DoH and improve its technical privacy guarantees instead of just asking people to trust them. Oblivious DoH[0] is an interesting proposal in that direction, there's also a decent amount of work (which Cloudflare is supportive of) going into research about by-default splitting DoH requests across multiple entities, so no single provider would get all of your queries. There are some obvious issues there to overcome regarding privacy, but it's still a somewhat promising proposal.

From a technical perspective, there is nothing about DoH that is centralized, and no part of the technology would force it to be centralized. There's no consumer lock-in, and there's no browser lock-in. Configuring DoH is not an "arcane configuration", it's just as easy as any DNS setup in your browser, even easier because there's less network interference and it's easier to debug.

But from a practical perspective right now on the ground, while I share concerns about Cloudflare overall and while I share concerns that Cloudflare consolidating power is problematic, I don't think it's accurate or reasonable to claim that DoH is a power grab. By default, Cloudflare isn't the provider in Canada. In the US, they have temporarily been chosen by the default by Mozilla because they are currently offering the best service. But anyone else could step into that position if they care to, there's no blocking or gatekeeping going on.

And frankly, if the only reason that DNS is decentralized is because most people's devices don't use a consistent resolver and just naively trust whatever network is were on, if the only reason why DNS is decentralized is because everyone's ISP sets it in the background without their knowledge, then in my opinion that's just a ridiculous way to guarantee provider diversity, and we should absolutely overhaul that system. Any system where my DNS provider changes behind my back when I connect to the wifi at a coffee shop is broken, and that is the experience that most nontechnical users have with DNS today. It is frighteningly insecure.

> You can't claim one minute that "we're doing this to take care of naive users who can't protect themselves", and the next minute that "users can set up any arcane configuration they want".

I don't think I do claim this. What I claim is that Cloudflare is an obvious default for nontechnical users, and I claim that DNS is susceptible to all of the same centralization concerns as DoH, and I cliam that changing a DoH provider in Firefox for most end users is no more difficult than it would be to change a DNS provider.

Most users won't change their DNS provider, which is why it makes sense to use a secure default rather than to let every random network in the wild decide. But if you are the type of person who wasn't just trusting network DNS, then DoH is not going to be a problem for you. It especially won't be a problem for you once OS manufacturers start allowing system-wide configs.

> If you suddenly take away the ISPs' visibility into many people's DNS, then the ISPs get a reason to care.

I buy this theory, I think you're very likely correct. But I don't think it's a compelling argument. If what you're saying is correct, then literally any privacy initiative that threatened ISPs in any way would be blocked. What's the takeaway from that, we should abandon efforts that threaten ISPs?

We are seeing a large amount of pushback from ISPs, from network operators, and from governments themselves over DoH. This does not suggest that the proposal is dead on arrival, or that it's just going to be globally blocked. If DoH didn't have teeth, we would not be seeing this reaction.

So what your argument is saying is that DNS skated under the radar largely because it wasn't helping anyone, and ISPs didn't feel threatened by it. But I'd rather throw my weight behind a technology that they do feel threatened by, rather than behind one that they don't care about. A world where we have to fight about DoH being blocked is still strictly better than a world where we don't have to fight because there's nothing to fight over and the majority of people are using a system that is trivially attackable.

[0]: https://blog.cloudflare.com/oblivious-dns/

For the record, I share your cautious confidence in Cloudflare. For an entity of its size, Cloudflare has been amazingly good both at being transparent and at actively finding ways that people don't have to extend any trust. Cloudflare even suggested real cryptographically blinded tokens as an alternative to CAPTCHAs (and saw it shot down by people who didn't understand the crypto, and were sure it was some kind of trap...).

Cloudflare could easily go bad in the future, but has been refreshingly good so far.

But most of the risks "native" to large commercial entities seem to have been using DNS queries to target advertising. That sucks, but it's not my top concern. I'm more worried about outside pressure to censor the Internet, which is more of a non-native risk. From my point of view, censorship is a bigger worry than targeted advertising. It can happen here, and it's already happening in a lot of "theres".

If some major government, or coalition of governments, or even major private pressure group, wants disappear something from the DNS, and there are only 15 or 20 TRRs, then it's a lot easier to lean on those TRRs than it would be to lean on hundreds of ISPs. In any given case, it may also be easier than leaning on the registrars.

Heck, since there's a centralized Mozilla policy for who gets to be a TRR, they can just lean on Mozilla. Maybe it won't matter because Chrome will have more market share or whatever, but it doesn't seem quite right for Mozilla to rely on that.

Even if Cloudflare happens to be much better about resisting filtering pressure than most ISPs, and stays that way, there's still only one Cloudflare that has to be subverted. And there's going to be pressure on Mozilla to include TRRs that might not even be as safe as Cloudflare.

You can already see the pressure to bake censorability into the TRR system. Look at the comments on the Mozilla consultation. You have a lot of people talking about how TRRs need to be able to (secretly!) respond to government blocking requests. You even have what looks like a coordinated push from UK commenters to protect secret non-government blocking at TRRs.

Yeah, we'll all disagree about whether X or Y should be censorable. Most people can name something that they think should be censored, and I think that a lot of people's lists are getting bigger rather than smaller. But once you start creating infrastructure for filtering, everybody comes out of the woodwork and starts trying to get their particular filtering priorities. It's hard for a centralized system to resist them.

The real answer is probably to switch to something a lot more decentralized and anarchic than the DNS can ever be. After all, in the end, there's only one ICANN. But that will take a very long time, if it ever happens at all. In the meantime, it seems really unsafe to concentrate power over the DNS in a limited number of pressure points.

"Who have shown repeatedly to be user hostile; especially towards privacy conscious users."

What examples were you thinking of.

They are still the only CDN offering TLS1.3 with ESNI. Problem is they do not necessarily encrypt backend connections. TLS version and SNI are relevant IMO because even with DoH, without TLS1.3 plus ESNI, the ISP can still see domain names in the clear on the wire in client hello packets sent and TLS certificates returned from remote IPs. All those AWS-hosted sites you visit all require SNI, the domain name sent in the clear over the wire for your ISP to track. For many, you can use the cloudfront CNAME as the SNI but it would be easy for an ISP to keep lookup table for all AWS CNAMES. HN's hosting provider will not even support TLS1.3, it's TLS1.2 at best.

SNI is hostile toward privacy conscious users and makes censorship easier.

But what’s the alternative to SNI for multiple domains on one IP? And don’t say IPv6, because my ISP doesn’t even offer that yet…

ESNI with TLSv1.3.

(The "E" means "encrypted".)

That does sound nice, but apparently it’s already been succeeded by TLS ECH, and neither of them are widely enabled by default, so essentially there is currently no SNI alternative.

ESNI (now called ECH): https://defo.ie

I use ESNI-enabled openssl from github.com/sftcd, 30 Aug 2019 release, for all HTTP requests to Cloudflare-hosted sites and it works well. A noticeable downside is one has to keep fetching a new key in a DNS TXT RR every hour. Another use for DoH; the key can re retrieved via DoH (no SNI required).

At this point. The vast majority of ISP dns servers are hostile. Here in Australia, the ISP dns servers are hostile by law so 100% are hostile.

If you are in the minority who have a great DNS server, go to the firefox settings and make a change. Defaults should suit the majority.

I note Firefox is turning this on country by country. Perhaps they could turn it on for certain countries (possibly including yours, Australia) but leave it off for others (like New Zealand where all ISPs don't interfere with DNS).

I don't know why Cloudflare is inherently more trustworthy than your ISP.

> There's a lot of lament that this bypasses the operating system level config, but I feel like we wouldn't be here if the operating system had stepped up their DNS any time in the last several years.

OSes are relegated to support myriad environments with adhoc yet byzantine configuration. A browser's primary objective is to be a "user-agent" (as opposed to, say, subscribing to the whims of ISPs / Governments / anti-digital-rights-companies-masking-as-cyber-security-firms).

> For the techie user who's running PiHole or some custom local DNS, yeah this will be annoying, but for the average user who's just delegating to their ISP that is then tracking their DNS lookups and injecting ads if they mistype a domain, this is a win.

Internet censorship, which most of the population in the West isn't subject to or even aware of, is the primary reason DoH-in-a-browser is widely lauded. It just works, and thwarts all cheaper attempts at surveillance and censorship. Though, the likely effect would be, deep-packet-inspection becomes common place, but the imminent ESNI standard has that one covered, too.

As someone who builds and maintains an open-source DoH resolver, I have come to fully appreciate the control and power a resolver running over HTTP/S puts in the hands of the users, both in the client and on the resolver-side.

> the imminent ESNI standard has that one covered, too.

Too slow, I'm afraid. Word on the street is that any hello containing ESNI will be blocked from day one in too many places for it to ever get traction, including all of China. I believe draft ESNI is blocked by the GFOC today.

To even have a chance of making ESNI/ECH widespread, you'd have to make it mandatory for all TLS. That wouldn't fly politically in the working group, because the "enterprise" people have at least some influence there, and they want to do exactly the same kinds of DPI that North Korea would want to do. Even if it came out mandatory on the standards track, it would probably still fail at this point.

It might have worked if it'd been mandatory in TLS 1.2 or even TLS 1.3, but it seems too late now.

We'll see if they also disrupt DoH to the point of making it unusable. I bet they at least give it a try.

If Microsoft, Apple, and Google all agreed to make ECH mandatory on the same future date, wouldn't that basically force the hands of every other enterprise?

The goal of ESNI is not to bypass China's intranet. The goal is to say to DPI countries such as the UK: either you implement a China-style great firewall or you implement a free internet. There is no middle ground where you can spy on people without it being obvious.

> That wouldn't fly politically in the working group, because the "enterprise" people have at least some influence there, and they want to do exactly the same kinds of DPI that North Korea would want to do.

If this was true, TLS 1.3 would have RSA kex. The "enterprise" people (specifically in fact EDCO, the Enterprise Data Centre Operators, Nalini Elkins' group) really wanted to keep RSA kex. It was removed in TLS 1.3 anyway because it's a bad idea.

In practice the intention is that ECH will be GREASEd, so yes, it will go from being nowhere to being apparently everywhere almost overnight. It's not specifically intended that GREASing ECH will defy the Great Firewall, but it's a possible outcome. China can decide its own policies without help from us.

For EDCO and similar outfits, ECH changes nothing.

> If this was true, TLS 1.3 would have RSA kex.

I hope you're right...

> In practice the intention is that ECH will be GREASEd

If even fake ECH extensions get rejected, then it's hard to deploy the whole GREASy burrito.

> China can decide its own policies without help from us.

... but China can make it uncomfortable to support ECH worldwide. It's not a free action for a service to risk being blocked in China, or for client software to accept being unreliable or unusable in China. That affects deployment in the rest of the world. We'll see who wins the war of wills, but I'm afraid that the "Western" entities that have the most control over what's widely deployed are not going to have will enough.

Any "service" that wants to support China needs to have serious presence there anyway.

They actually are recommending for everyone to add fake ESNI/ECH data even if you don't intend to use ESNI/ECH.

See: https://datatracker.ietf.org/doc/html/draft-ietf-tls-esni#se...

I'm afraid browsers stopped being "user agents" a long time ago. It's one of my pet peeves. They're agents of whichever company rolls them out. Basically Google or Mozilla. And they defer to the whims of whoever built the website you're visiting for a good chunk of their (insane) behaviour.

If they were actual user agents, you would still have control over a lot of behaviour that's now been removed. Like actually remembering form fields you want to remember (instead, some websites tell the browser that it can't remember the user name, and the brower foolishly obeys). Or like copy/pasting text. Or like using the right mouse button (some websites disable it). Or like having control over keyboard shortcuts, which some websites override and which can't trivially be fixed.

In my book, a user agent acts in my interests and on my behest. Sure, most of these behaviours can be corrected using browser extensions, but requiring a boatload of extensions to have an actual "user agent" doesn't make me happy.

>There's a lot of lament that this bypasses the operating system level config

systemd-resolved has the clandestine ability to forward all your DNS traffic to google in many distributions of Linux. I would say the state of resolution in general is getting uncomfortably sketchy.

many wireless routers from Asus and others due to a buggy (or perhaps intentional) implementation of Dnsmasq will quietly intercept your DNS requests using their own built-in resolver. RT-ARCH13 is notorious for it even if NAT mode is disabled.

for my pihole implementation I block 53 outbound and do DoH resolution using cloudflared on the pi.

finally it bears remembering that the number one largest opposition to the use of DoH was ISP's who were (and still are) making a killing selling your search history. they went so far as to lobby congress to stop DoH. Now the very same ISP's we tried to protect users from are part of the DoH club anyway because, well, money talks.

https://arstechnica.com/tech-policy/2020/06/comcast-mozilla-...

I recently tried setting up microk8s, since Ubuntu was pushing it pretty hard and it seemed useful, but a lot of stuff wouldn't work at all.

Turns out that the microk8s config uses cloudflare DNS by default, and not your currently configured DNS or any kind of local DNS proxy. This was hugely frustrating because it's not the configuration that I would expect, it's not obvious why things are failing, and, in our case, it's sending internal DNS names to an external DNS server, unencrypted, over the public internet.

Surely it's convenient, but definitely not trustworthy.

"clandestine"? Please, don't be ridiculous. The choice of fallback DNS server is a default that is only used in the absence of any other configured DNS resolver. If you are compiling systemd yourself then this is one of many defaults that you need to be aware of. Everyone else will get the defaults that their distribution has chosen.

If Mozilla cared about preventing DNS tampering, they could have done DNSSEC 10 years ago. That would have totally killed ads, ISP typosquatting, and captive portals. They, like all browsers, rejected that source of end-to-end security.

The fact is that they want to enable DNS tampering, because they want to use it as a way of enforcing their idea of "safety". So they're building this giant, highly centralized, and easily pressured, infrastructure of "trusted resolvers" that can mess with your DNS results to their heart's content. This will, of course, be a huge attractant for bad actors who want to do things even more harmful than injecting obnoxious advertizing.

As for privacy, the only way that can work, even with something like DNSCrypt, is if the individual user chooses the resolver from among a very large set of possibilities.. which Mozilla is not enabling. Either that, or relatively exotic anonymization stuff that's not actually worked out, and that Mozilla has shown zero interest in working out. "Trusted resolvers" just move the privacy problem to a different place, and, again, it's a highly centralized place that's easy to subvert.

And DNS-over-TCP is desirable for sure, but it does nothing about tampering or privacy. It's just irrelevant, except maybe as a preliminary step making it easier to deploy some of the stuff that might actually help.

How could Mozilla have "done" DNSSEC 10 years ago? Isn't that pretty much just not in their control?

If TLDs are signed (which they have been for a long time), and if the browser checks DNSSEC, you can't typosquat, because you can't spoof the TLD server's signature on the typo. That requires no cooperation from anybody but the browser and the TLD.

WIth DNSSEC enabled, somebody can still impersonate an unsigned domain, but the whole reason that there are so many unsigned zones to begin with is that nothing checks the signatures, so there's no incentive to sign. By actually doing the checks, browsers, including Firefox, could have created an incentive for widespread DNSSEC signing.

So they could have both unilaterally gotten a lot of benefit for their users, and encouraged wider DNSSEC deployment that would have produced further benefit.

Add in DANE, and it would also have given a much more satisfying fix for a bunch of CA issues that they've chosen to band-aid with stuff that only half works. You could probably even have gotten to a world with zero-cost TLS certs well before enough pressure built up to make Lets Encrypt come along.

Interesting, thank you. That does sound more satisfying as a fix (though also sounds harder and possibly riskier. If they're the only ones checking, will some sites be broken only in Firefox?)

You're saying it's one way, but it's the opposite way. Huge portions of the Internet's user base resolves through servers that will validate DNSSEC if it's available. Still, nobody --- most especially not organizations with serious security teams --- signs their domains. That's not because they don't care about security. It's because DNSSEC makes sites unreliable, and the failure mode for DNSSEC is just about the worst possible mode: your site simply vanishes from the Internet, as if it hadn't existed at all.

None of this is hypothetical. It happened, for instance, at the launch of HBO NOW, when a screwed up DNSSEC record made the site vanish for everyone resolving through Comcast, which pointlessly verifies DNSSEC.

DNS without DNSSEC will also "make your site vanish" if you administer it incompetently. And that's not hypothetical either; it happens all the time.

As for the "servers that will validate DNSSEC if it's available", the operators of the servers that "huge portions of the Internet's user base" use are exactly the people who are potential threat actors in most DNS tampering scenarios. That's not useful or effective validation. It makes no sense to put the choice of whether to validate in their hands, and obviously nobody can trust their validation.

DNSSEC does not protect privacy which is more important than integrity since TLS already does that.

Even worse, making DNSSEC well used might have pushed privacy protocols out of priority as people claim DNS is good enough already.

One of the reasons Mozilla couldn't "do" DNSSEC 10 years ago over plaintext DNS is that for some (enough) clients this doesn't work. Their DNS is broken, it was always broken, but so long as you only ever ask something easy like "news.ycombinator.com A?" it worked well enough. Asking DNSSEC queries doesn't work. "Last Change Broke It" rules apply so Mozilla gets the blame.

With DoH of course this is much less relevant. The people operating Trusted Resolvers for DoH are obliged to actually implement the protocol correctly.

If you choose a DoH provider which does DNSSEC for you (such as Cloudflare's 1.1.1.1), and you trust that provider, your answers are DNSSEC validated, today, without any changes to Firefox. Millions of people benefit, zero work.

If you're concerned that Trusted Resolvers aren't enough privacy, Oblivious DoH eventually means that the TRs don't see your query any more, so the people answering the query don't know who you are, and people who know who you are don't know what your query is.

I'll bet a comparable number of clients won't be able to reach DoH servers for one reason or another. Especially because there seem to be people who are going to actively try to block them. Whenever you change anything, something will break.

And it would have been relatively easy to put up a notice that said "your DNS server seems to be dropping all EDNS0 data or whatever stupid thing it's doing; would you like to switch servers"? Which would have avoided centralizing everybody by default.

I know there were enterprise firewalls that broke DNSSEC on any packet that went through them... but frankly the sellers of those firewalls deserved to get heat, and any enterprise that would deploy that kind of broken crap isn't an organization I'd want to support as user.

And that still doesn't justify putting DNS on top of HTTP of all things. But that's another story.

As for Cloudflare doing DNSSEC checks, the fact that I shouldn't be trusting Cloudflare to do the validation for me is kind of the point. Validation (final validation, at least) belongs in the endpoint.

If "oblivious DoH" is what it sounds like, it sounds like maybe I have to retract "zero interest in exotic anonymity". Which I am very happy to have to do.

DNSSEC has never validated at the endpoints. By design, validation occurs --- has to occur --- at the recursive server. The channel between you and your recursor is "secured" with a single bit, which says "yeah, sure, I did DNSSEC validation". Don't blame the systems software for this; it's a terrible protocol.

I am typing this on an endpoint that does DNSSEC validation.

You can also run default-free full-feed BGP4 on your endpoint, if you're just trying to make a point.

I don't run it to make a point. I run it because it's relatively easy to set up even with the presently available software, and it improves my confidence in what I get from the DNS. It would be even easier to set up on an endpoint if DNSSEC were something that people providing OSes cared about. In fact, there's no reason that setting it up intrinsically requires any user interaction at all; it could easily be the default.

You claimed that it "by design, validation occurs -- has to occur -- at the recursive server". That is false, and it is false in a way that's important to the issues under discussion. Nothing forces validation to happen at servers other than laziness and inertia.

I'm not sure you understand what I'm saying. DNSSEC validation occurs step by step through the recursive resolution process. You can't simply request records from a recursive server and then DNSSEC-validate them yourself. I don't think DNSSEC works the way you think it does.

Basically the message here is "DNSSEC will work when customer DNS servers go away and everyone just runs their own". Ok, I'll add that to the (long) list of obstacles to its adoption.

I understand exactly how DNSSEC works. It's been a while since I was really close to it, so I doubt I could sketch the packet formats any more, but I'm crystal clear on the structure... and I still admin it for multiple zones.

Insofar as I can extract any meaning from what you say, what you say is false. DNSSEC validation does not take place "step by step through the recursive resolution process". You seem to be conflating the resolver tree with the zone tree.

Any entity that can get access to the necessary DS, KEY, and RRSIG records (plus the actual data records) can do a complete local DNSSEC validation from the root to the leaf. It can get those records from anywhere. It doesn't even have to get all of them from the same place. It doesn't have to rely on any validation from the upstream servers, and is in a position to independently verify any validation they do do.

In fact, that's a basic DNSSEC design feature. One of the design goals was that intermediate responders didn't have to know about DNSSEC at all. Otherwise it would have been impossible to ever deploy it.

The protocol provides for asking a responder not to return invalid records, or even to return them but flag them as suspect. Clients don't have to use that functionality. Clients can always ask for all of data and for themselves. And any responder that does set any validation-related bits is expected to have verified the entire signature chain for itself, not deferred to any validation the next upstream responder claims to have done.

There's nothing "step by step" about it.

It's true that DNSSEC validation at the application end is <em>typically</em> done by a stub resolver, by which I mean a specific kind of limited DNS server that does recursive lookups and caches results for some defined population of clients, but is not authoritative for any zone. Unfortunately the word "resolver" is confusing, because it's used to refer both to stub resolvers and to more or less any software that does DNS lookups at a client, including libraries linked into application programs.

Nonetheless, the validating entity does not have to be a stub resolver or any other kind of DNS responder. It can be a "resolver" in the sense of a library, or it can be something else. Applications don't have to use the DNS protocol to talk to it, and if they do, they can do so entirely locally on a single host.

But you can still do on-endpoint validation using a stub resolver, just by using a local one that serves only applications on a single host.

To head off your next load of FUD about "running your own DNS server", such a local stub resolver is a small thing that can be installed by default, because you can get it working with zero configuration or administration, beyond maybe a forwarder address that it can get from DHCP.

Running an on-node caching resolver is nothing like running an authoritative DNS server, or even a stub resolver serving a whole network, and I know very well that you know that. Don't try to compare them.

A local resolver process doesn't demand a lot of resources by post-1990s standards, and the end user doesn't even have to know it's there. In fact, operating systems typically have similar local server processes anyway, for performance reasons. That's exactly what nscd or systemd-resolved does. Indeed, systemd-resolved, even though it was introduced for unrelated reasons, can be told to do DNSSEC validation, removing any need at all for a separate DNS-based stub resolver process.

I have trouble believing you're confused about any of this. What you're writing here, and what you've written elsewhere, sounds too much like the output of somebody trying to generate FUD that will convince the half-informed. And, yes, I did read that enormous screed you wrote a few years ago, with all the other gross, and suspiciously convenient, misunderstandings and mischaracterizations.

This is a long description of how an endpoint can do much of the work a recursive cache does for itself. We all understand that it can. You've clarified, you understand how DNS servers do lookups; I won't suggest that you don't again. It was never my claim that every server in the DNS topology needed to be DNSSEC-aware (though, obviously, every authority from leaf to root needs to be).

You haven't made any headway in your argument about how everyone is going to run DNS software that does this stuff. It's "easy" to do this stuff, in the same sense that it's trivial to run a local djbdns instance on your laptop instead of relying on your ISP's server. My point is: that's not how DNS is normally deployed.

> And it would have been relatively easy to put up a notice that said "your DNS server seems to be dropping all EDNS0 data or whatever stupid thing it's doing; would you like to switch servers"? Which would have avoided centralizing everybody by default.

Would you really expect ordinary users to understand that and respond appropriately?

> If Mozilla cared about preventing DNS tampering, they could have done DNSSEC 10 years ago.

DNSSEC only protects integrity, not confidentiality or availability. It still lets evil ISPs build a profile of every domain you've ever resolved, and to block your access to certain domains by just dropping your DNS request packets for them. Basically the only thing it stops is them sending you to a fake page (e.g., a pretty block message rather than a generic browser error).

Also no one cares about integrity since it is already validated by TLS. And it seems that DNSSEC is not even widely supported by TLDs. My domain says DNSSEC is not available.

Seems like DNSSEC is another failed spec like certificate pinning.

We need DNSSEC for eSNI/ECH to be possible.

The eSNI drafts I read say explicitly that DNSSEC is not required (there's a whole section about it), which is good, because if it were, eSNI would be dead in the water.

I looked again and you're correct. For other readers, the justification is basically that if an attacker can MITM DNS, then they can find out what domains you're going to even without compromising eSNI/ECH.

I double checked the RFCs, but I'm not kidding that you can generally work this stuff out axiomatically (for a change) --- if an important new standard requires DNSSEC, it's not an important new standard, because it can't happen. DNSSEC is moribund.

> This is a workaround for the fact that many networks are hostile to DNS lookups (injecting ads, tracking, captive portals), and operating systems continue to just default to the ISPs DNS provider, which is often hostile.

This workaround is hostile to my own network (home or work) being able to track requests. I control my router / DCHP server, and I should be able to tell devices where to send DNS requests.

Anything that does not do what I tell it to do is malware IMHO.

You can’t track all requests anyway, as they can be trivially tunneled through encrypted streams with bundled certs.

This is the tradeoff of running arbitrary Turing-complete programs. If you don’t like what a program is doing, or that it doesn’t use libc or some standard way to do DNS resolution, your only options are to either change the program to do what you want, or don’t run it.

> You can’t track all requests anyway, as they can be trivially tunneled through encrypted streams with bundled certs.

Previously I can track devices (a) making DNS requests, and then (b) make TCP and/or UDP requests. Anything that made a data request without a DNS request was probably using a hard-coded IP, which would be suspicious.

With DoH, it's all-HTTPS all-the-time:

> DoH is an over the top bypass of enterprise and other private networks. But DNS is part of the control plane, and network operators must be able to monitor and filter it. Use DoT, never DoH.

* https://twitter.com/paulvixie/status/1053886628832382977

> or don’t run it.

So Firefox further circles the drain with regards to market share? Is this part of some grand 4D chess plan by Mozilla that is perhaps beyond my intellect?

The cat is out of the bag regardless of what Firefox does. Even without firefox any bit of malware could bundle their own DoH library.

> and I should be able to tell devices where to send DNS requests

But you can't. Even if DoH went away today, even if Mozilla and everyone else dropped the technology, you still don't actually have the ability to do that. Devices will still be able to ignore you.

It's not DoH's fault if devices on your network don't use your network-wide DNS resolver. IOT devices were going to move in that direction regardless. Increasingly, your smart TVs are going to ignore your network settings. They might not even make DNS requests over the correct port. They might hide them in normal Internet traffic. They might cache them locally. They might make a request for a domain that looks safe, and then use a different IP address that they have cached locally. They might even ship with their own embedded 5G chip and use it for specific exfiltration requests and updates. They might join your local network, read all of your unencrypted traffic, and then use a 5G chip to separately send data out.

If you think that blocking DoH will solve these kinds of problems, I feel like you are putting way, way too much faith into your existing DNS setup.

> Anything that does not do what I tell it to do is malware IMHO

Okay, that seems completely reasonable. However, my perspective is that any network that will MITM my requests or monitor what websites I visit without my permission is malicious IMHO. It sounds like the answer here is to block access to your network for devices that you don't trust, or to try and quarantine them -- not to demand that my personal device stop respecting my own entered DNS settings whenever it joins a new network.

To argue that my installation of Firefox or that my OS shouldn't be able to ignore network settings anywhere and that I shouldn't be able to choose how my requests are encrypted, just because it makes it easier for you specifically to enforce your network's security policy without explicit opt-in or consent -- I don't know, it just seems pretty backwards to me from how this kind of security stuff is normally supposed to work.

I know that this stuff will make it harder for some network operators to do their jobs but... so did HTTPS. I heard literally the exact same complaints during the early push to get every website encrypted. But we live with it, and now if you want to monitor my unencrypted network traffic, you need to get my permission to install a certificate. Control and consent slowly moves away from networks and onto devices over time, closer to the users themselves.

> Devices will still be able to ignore you.

Yes. But previously by logging requests to port 53 (and 853) I can hunt down those devices quite easy.

> However, my perspective is that any network that will MITM my requests or monitor what websites I visit without my permission is malicious IMHO.

If you don't like how I've set up my network, blocking your DNS requests, feel free not to use my network.

> To argue that my installation of Firefox or that my OS shouldn't be able to ignore network settings anywhere and that I shouldn't be able to choose how my requests are encrypted, just because it makes it easier for you specifically to enforce your network's security policy without explicit opt-in or consent

You can set up your devices however you want. But if they're on my network, they play by my rules.

> I know that this stuff will make it harder for some network operators to do their jobs but... so did HTTPS.

And network owners are free to block port 443 as well. Or any other port. People can choose to use those networks or not.

> But previously by logging requests to port 53 (and 853) I can hunt down those devices quite easy.

And IOT devices are required to use port 53 or 853? They don't have the ability to proxy their DNS requests through another service? DNS even as it exists today is not a mandate, this stuff can be circumvented.

> You can set up your devices however you want. But if they're on my network, they play by my rules.

I think that's fine. Block DoH on your own network. But don't complain about my browser turning it on. You can block 443 if you want, but I'm not going to be sympathetic if you argue that LetsEncrypt shouldn't exist.

There's a big difference between saying that you require every device on your network to plain-text its DNS queries to you for monitoring, vs arguing that it's harmful for Mozilla to make Firefox's DNS queries more secure for everyone else.

The only way DoH would be hostile to your network is if needing consent to monitor people's DNS queries when they join your network (via a settings change or confirmation that they're not using DoH) is a problem for you.

DHCP is a mechanism to automatically configure clients with useful defaults, like a DNS server. It's not a system for enforcing network policies. There was never supposed to be any requirement to observe what DHCP tells you.

I didn't realize that Fedora dropped that proposal https://pagure.io/fesco/issue/1511

May have been related to IPv6? Not sure.

I remember trying it out at the time (https://fedoraproject.org/wiki/Changes/Default_Local_DNS_Res...) and not running in to issues

Having had the joy of writing code to manage system wide DNS config for all 3 of the major desktop operating systems, it's very clear that nobody cares about DNS - the users just want it to work and the systems developers also just want it to work. On macOS, if you don't have an IPv6 enabled primary adapter, even if you have a working IPv6 VPN connection, you just won't be able to get AAAA responses regardless of your configuration. Watching for config changes is also a race ridden mess, applying DNS on Linux on major distros is also a crapshoot of which specific NM (NetworkManager) version the user is using (this can differ between a fresh Ubuntu 16.04 install and one which has been upgraded to 16.04 from 14.04), and which specific DNS mode NM is configured for. There's also systemd-resolved, with multiple versions and capabilities, and there's also resolvconf. In some cases, there's a notion of a global config, in others, the config is always bound to a specific interface. Windows is somewhat better in this regard. But at least the developers of NM and systemd-resolved are receptive and incredibly helpful.

How OS is going to circumvent local DNS? Hardcode 1.1.1.1? It's very easy for ISP to block this address.

Release (not beta) macOS already lets you configure your system to use DNS-over-HTTPS, if you know the magic incantation to chant at it. Apple hasn't added UI for it yet, but it's available and works when enabled.

> ISPs DNS provider, which is often hostile

Exactly which Canadian ISP does this? I don't think that's an issue in Canada, where this feature is being enabled.

On Linux there is systemd-resolved which is enabled installed and enabled on the most popular distros. It is a caching resolver that supports DoT (although it isn't enabled by default).

Having the pre-select a DNS resolver some of the same problems as having the application selecting the resolver. Specifically, it doesn't work as expected in networks with a local DNS server.

This is where iOS wins I think. According to Apple documentation, no app can override a systemwide DNS setting in iOS14. So if you were to apply a dns profile, it is guaranteed that apps cannot use their own dns resolver.

Which documentation? I searched the App Store Review Guidelines for "DNS" and didn't find anything.

They have, will Firefox respect the settings made there?

ie Windows 10 has it, macOS, Linux with systemd-resolver has it, Android too.

Mozilla wants to be the gatekeeper to their users. Every company and it's dog wants to do this. "If you are not paying for the product, you are the product" should become "if you are a customer/user, you are also a product".

Reply on news.ycombinator.com