marcusb

> do you still pay pay Ente if you self host the server as well as the photos ("S3-compatible object storage")?

No. (I self-host Ente and use their published ios app.)

Also a wonderful zine!

> The TTL for NSEC records are presumably way lower than the TTL for the DS records.

Possibly. It was still an outage they had to wait out the TTL for, due to the design of DNSSEC.

> It’s theoretically possible that it would not have worked for all cases, but that is, in my experience, very unlikely.

This is completely unsubstantiated speculation on your part.

> The bug seems to me to have been reasonably easy to mitigate, but their problem was that Slack did not know what they were doing.

It is indeed reasonably easy to Monday-morning quarterback someone else's outage and blame operators for the sharp edges around poorly designed protocols.

> 1. That number used to be zero, as tptacek liked to point out.

Cool. so, at this rate, in another 100 years or so we should be at 50% adoption.

> 2. The huge operators often have fundamentally different security priorities than regular companies and users.

Priorities like uptime?

> 3. People said the same about IPv6 and SSL, which were also very slow to adopt. But they are all climbing

1) people started rolling IPv6 out once v4 addresses got scarce. There is no such compelling event to drive DNSSEC adoption. 2) SSL is easy to roll out and provides compelling security benefits. It is also exceedingly unlikely in practice to blow up in your face and result in run-out-the-clock outages -- unlike DNSSEC.

> You don't get that under regular contracts either.

Absolutely false. Of course vendors sometimes mark things WONTFIX, but Microsoft regularly produces bugfixes for supported products based on issues identified in support cases... As does every other reputable software vendor.

> EOL either means "no more fixes period" or means nothing.

Well, I disagree. Can you call in and get support with a support contract? Can you get a support contract without a one-off negotiation? Does the vendor regularly produce bug fixes -- not just emergency security fixes to allay a PR disaster -- for the product? No to all three? EoL.

Most important of all, has the vendor signaled that they will not support the product after X date and therefore a customer without a bespoke contract cannot rely on said support? EoL.

No. If a vendor says “we aren’t going to support this product after X date - don’t call, don’t write”, that’s EoL.

Doesn’t matter if they do one off fixes because they decide that’s the right thing to do - product is still EoL. You won’t get support if, say, Word crashes due to a core library bug. You can’t rely on them doing regular testing. EoL.

Doesn’t matter if the DoD comes to some ridiculously expensive bespoke support arrangement - still EoL. You could probably offer them enough money to provide a support contract for MSDOS 1.0, but that’s still EoL for everyone else and in general.

> Microsoft has a long history of playing fast-and-loose with the truth. And that’s again the case with Windows 10 coming to its supposed “end of life” this fall.

I can’t take an article seriously, whatever merits it might have, if this is the opening gambit.

“End of life” is a fairly common term of art amongst software and hardware OEMs. Windows 10 is going to be end of life. No scare quotes needed.

I imagine it depends on which "top domain" list you use. I use Cloudflare radar. As of today, for their top 100 domains, 6 have published DS records:

2371 13 2 32996839A6D808AFE3EB4A795A0E6A7A39A76FC52FF228B22B76F6D6 3826F2B9 cloudflare.com

2371 13 2 F52DBA4AAEA13A1F457C0FB4C1953F40E16AFC5C5E79EDF7CEED0FCF 0CBD81F0 cloudflare-dns.com

53074 13 2 86F2929EE3E5E501032B6DC94841A4A056A2D2876CABCF46A5F8907E B4917782 one.one

56044 8 2 1B0A7E90AA6B1AC65AA5B573EFC44ABF6CB2559444251B997103D2E4 0C351B08 dns.google

48553 13 2 57AF2F182A541A91AD24CC6583867C2BA331255B03E2A32579A625AD 1F3BE3CA taboola.com

33751 8 2 90C6CD28626CA7B8E3A1FACAD58D20D486E52DF040B9B2F085ACD5C7 03E624C6 nist.gov

Not that it matters much one way or the other - you won't find a top 100 domains list where, say, 50 have DNSSEC enabled.

> They could have done a quick fix by adding an explicit app.slack.com record. But instead they removed the DNSSEC signing from the whole domain, thereby invalidating all records, not just the wildcard ones.

1) That would do nothing to fix resolvers that had already cached NSEC responses lacking type maps.

2) That presumes the wildcard record was superfluous and could have been replaced with a simple A record for a single or small number of records. Would love to see a citation supporting that.

3) That presumes the Slack team could have quickly identified that the problem they were having was caused by the fact that app.slack.com (and whatever other hosts resolve from that wildcard) was caused by the fact the record was configured as a wildcard and would have been resolved by eliminating the wildcard record. If you read the postmortem, it is clear they zeroed in on the wildcard record as being suspect, but had to work with AWS to figure out the exact cause. I doubt that was an instantaneous process.

Any way you slice it, there was no quick way to fully recover from this bug once they hit it, and my argument is that the design of DNSSEC makes these issues a) likely to happen and b) difficult to model ahead of time, while providing fairly marginal security benefit.

At this point, I really don't care if you agree or disagree.

> I will care once something else comes around with any promise of being implemented and rolled out.

Yeah. DNSSEC is going to be widely deployed any day now. The year after the year of Linux on the desktop.

> I work at a registrar and DNS hosting provider for more than 10.000 domains. More than 70% of them have DNSSEC.

Cool. There are, what, 750 million domains registered worldwide? We are at nowhere near 10% adoption worldwide, let alone 70%. Of the top 100 domains -- the operators you would assume would be the most concerned about DNS response poisoning -- *six* have turned DNSSEC on.

Agreed, but for this particular product, the number is probably a lot closer to zero than average.

They do validate signed zones -- that is true. How do they run their own zones?

  $ dig +short letsencrypt.org DS
  $

> IIUC, if Slack had done the correct thing, only wildcard DNS records (if any) would have been affected

There's the problem - you DON'T understand. Straight from the post-portem that you clearly have not read:

  One microsecond later, app.slack.com fails to resolve with a ‘ERR_NAME_NOT_RESOLVED’ error:

  [screenshot of error ]

  This indicated there was likely a problem with the ‘*.slack.com’ wildcard record since we didn’t have a wildcard record in any of the other domains where we had rolled out DNSSEC on. Yes, it was an oversight that we did not test a domain with a wildcard record before attempting slack.com — learn from our mistakes!

> I don’t care. So is almost every other standard,

Cool story. I do care. I'd like to see greater protection of the DNS infrastructure. DNSSEC adoption is hovering around 4%. TLS for HTTP is around 90%. At least part of that discrepancy is due to how broken DNSSEC is.

Having read the post mortem, I disagree. Slack engineers did something dumb, under pressure during an outage. Even if they hadn't, they still would have been in a degraded state until they could properly remove their DNSSEC records and/or get the Route53 bug they hit fixed. In other words, they still would have had a 24+ hour outage, albeit with a smaller blast radius.

The design of DNSSEC is simply not fit for purpose for zone operators. It is far too easy to screw your zone up for far too marginal a benefit, to say nothing of the huge increase in CPU resource required to authenticate DNSSEC record chains.

The story for implementers is just as bad - the specifications are baroque, filled with lousy crypto and poorly thought-out options.

To give just one example, consider the NSEC3 iteration limit field. NSEC3 itself was designed mostly[0] to prevent zone enumeration when validating negative responses (which is trivial to perform with NSEC.) The iteration count was designed to give zone operators the ability to increase the cost to an attacker of generating a dictionary of nsec3 names[1]. Of course, a high iteration count also raises the cost to a non-malicious resolver of validating a negative response for a nsec3-enabled zone.

In good old DNSSEC fashion, the iterations field is a single number that is subject to a... wide variety of potential limits:

* 16 bits by the wire protocol

* 150 for 1,024 bit keys (RFC 5155 10.3[2])

* 500 for 2,048 bit keys (RFC 5155 10.3[2])

* 2,500 for 4,096 bit keys (RFC 5155 10.3[2])

* 0 (RFC 9276 3.2)

Why 0? It was noted -- after publishing the NSEC3 spec -- that high iterations just don't provide that much benefit, and come with a high cost to throughput. Appendix B of RFC 9276 shows a roughly 50% performance degradation with an iteration count of 100. So, RFC 9276 3.2 says:

  Validating resolvers MAY also return a SERVFAIL response when processing NSEC3 records with iterations larger than 0.

Of course, their guidance to implementers is to set the limits a bit higher, returning insecure responses at 100 iterations and SERVFAIL at 500. That said, if you want to be maximally interoperable, as a zone operator, you should pretend like the iteration count field doesn't exist: it is standards compliant for a validating resolver to refuse an nsec3 response with more than a single hash round.

As I said, this is one example, but I'm not cherry picking here. The whole of the DNSSEC spec corpus is filled with incomprehensible verbiage and opportunities for conflicting interpretations, far beyond what you see in most protocol specs.

0 - also to reduce the size of signed top-level zones

1 - all NSEC and NSEC3 records, while responsive to queries about names that don't exist, consist of obfuscated names that do exist.

2 - According to the letter of the standard, the limits applied to the iterations field should be 149, 499, and 2,499. Implementations are inconsistent about this.

The primary DNSSEC standards, RFC 4033-4035, are 20 years old. It isn't "in its infancy."

That is an extremely uncharitable take for an outage that involved two of the most poorly defined and difficult to (correctly) implement DNS features (wildcards and NSEC records) that the three major DNS operators mentioned in the Slack post-mortem (AWS, Cloudflare, and Google) all implemented differently.

The API was horrible, though.

It's worth it - just check to see if any big events are going on in Gatlinburg before you book. We went there one time (without checking) during a hot rod/motorcycle convention. The traffic was insane.

Back in the 90s, a portion of Germany's ATC ran on Emacs Lisp. No floppies though.

https://old.reddit.com/r/emacs/comments/lly7po/do_you_use_em...

We also have a lot of systems (references, the tort system) that just don't apply in any practical way to LLM output. I mean, I guess you could try to sue Anthropic or OpenAI if their chat bot gives you bad advice, but... good luck with that. The closest thing I can think of is benchmark performance. But I trust those numbers a lot less than I would trust a reference from a friend for, say, a plumber.

I understand a lot of people use LLMs for things they don't understand well. I just don't think that is the best way to get productivity out of these tools right now. Regardless of how people may or may not be used to outsourcing things to other humans.

> why would it be different with LLMs?

Because LLMs are not competent professionals to whom you might outsource tasks in your life. LLMs are statistical engines that make up answers all the time, even when the LLM “knows” the correct answer (i.e., has the correct answer hidden away in its weights.)

I don’t know about you, but I’m able to validate something is true much more quickly and efficiently if it is a subject I know well.

I’m puzzled when I hear people say ‘oh, I only use LLMs for things I don’t understand well. If I’m an expert, I’d rather do it myself.’

In addition to the ability to review output effectively, I find the more closely I’m able to describe what I want in the way another expert in that domain would, the better the LLM output. Which isn’t really that surprising for a statistical text generation engine.

> For the nth time, though, I also don't grant the premise that LLMs are violative of the IPR of programmers.

I wish you all the best waiting for a future where the legislature and courts decide that LLM output is violative of copyright law only in the visual arts.

> I just don't want to hear any of this from developers.

Well, you seem to have posted about the wrong topic in the wrong forum then. But you’ve heard what you’ve wanted to hear in the discussion related to this post, so maybe that doesn’t really matter.

> Adobe is one of the most successful corporations in the history of commerce; the piracy technologists enabled wrecked most media industries.

I guess that makes it ok then for artists to pirate Adobe's product. Also, I live in a music industry hub -- Nashville -- you'll have to forgive me if I don't take RIAA at their word that the music industry is in shambles, what with my lying eyes and all.

> Again, the argument I'm making regarding artists is that LLMs are counterfeiting human art. I don't accept the premise that structurally identical solutions in software counterfeit their originals.

I'm aware of the argument you are making. I imagine most of the people here understand the argument you are making. Its just a really asinine argument and is propped up by all manner of special pleading (but art is different, programmers are all naughty pirates that deserve to be punished) and appeals to authority (check my post history - I've established my bona fides.)

There simply is no serious argument to be made that LLMs reproducing one work product and displacing labor is better or worse than an LLM reproducing a different work product and displacing labor. Nobody is going to display some ad graphic from the local botanical garden's flyer for their spring gala at The Met. That's what is getting displaced by LLM. Banksy isn't being put out of business by stable diffusion. The person making the ad for the botanical garden's flyer has market value because they know how to draw things that people like to see in ads. A programmer has value because they know how to write software that a business is willing to pay for. It is as elitist as it is incoherent to say that one person's work product deserves to be protected but another person's does not because of "creativity."

Your argument holds no more water and deserves to be taken no more seriously than some knucklehead on Mastodon or Bluesky harping about how LLMs are going to cause global warming to triple and that no output LLMs produce has any value.

I generally agree with your post. Many of the arguments against LLMs being thrown around are unserious, unsound, and a made-for-social-media circle jerk that don't survive any serious adversarial scrutiny.

That said, this particular argument you are advancing isn't getting so much heat here because of an unfriendly audience that just doesn't want to hear what you have to say. Or that is defensive because of hypocrisy and past copyright transgressions. It is being torn apart because this argument that artists deserve protection, but software engineers don't is unsound special pleading of the kind you criticize in your post.

Firstly, the idea that programmers are uniquely hypocritical about IPR is hyperbole unsupported by any evidence you've offered. It is little more than a vibe. As I recall, when Photoshop was sold with a perpetual license, it was widely pirated. By artists.

Secondly, the idea -- that you dance around but don't state outright -- that programmers should be singled out for punishment since "we" put others out of work is absurd and naive. "We" didn't do that. It isn't the capital owners over at Travelocity that are going to pay the price for LLM displacement of software engineers, it is the junior engineer making $140k/year with a mortgage.

Thirdly, if you don't buy into LLM usage as violating IPR, then what exactly is your argument against LLM use for the arts? Just a policy edict that thou shalt not use LLMs to create images because it puts some working artists out of business? Is there a threshold of job destruction that has to occur for you to think we should ban LLMs use case by use case? Are there any other outlaws/scarlet-letter-bearers in addition to programmers that will never receive any policy protection in this area because of real or perceived past transgressions?

> Well that just doesn't sound true at all.

That would be because it is not true.

I must be the luckiest person in the world, then. The three I purchased have been installed for five years now, with zero leaks.

I bought $25 versions of this on Amazon during COVID... We did not anticipate or have any of problems you described.

The styling of this reminds me of faces that cant and/or serif the hyphen. I checked a few, and in my very limited survey (Blado, Poliphilus, Trajanus and "related" faces for each that Monotype recommends) most type faces that style the hyphen do not change the appearance of the emdash at all from a horizontal line. The few counter-examples I found are italic faces (Balladeer by URW.)

Yes.

just go and look at any of the vendors selling “zero trust” solutions. They all have white papers available about how a) perimeter security is “dead” and b) how their specific flavor of zero trust is the One True zero trust and the only thing you can trust to protect your data.

You will without exception need to provide an email address to access these white papers, so their inside sales team can ensure you fully understand the importance of trusting their zero trust, and not trusting anyone else’s.

I’m not kidding - even a little.