Hacker News Reader

Comment by ale42 | Hacker News Reader

The official definition of SHOULD per RFC2119:

  3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
     may exist valid reasons in particular circumstances to ignore a
     particular item, but the full implications must be understood and
     carefully weighed before choosing a different course.

Not sure how the people at Google interpreted this about the message-id

View on HN

You can argue that you not obligated to use message-id but if you don't use it you should blame only yourself that your messages are not accepted. In requiring message-id I would side with google (though in general I think they anti-spam is too aggressive and lacks ways to report false positives). Full RFC compliance (as in not only MUST but also SHOULD unless you have a very good reason) is the easiest part of making sure your emails will be delivered.

> if you don't use it you should blame only yourself that your messages are not accepted

I think it's a gray area

- If the receiver declines your message because "Message-id" is required - then I blame the receiver; because that's not true

- If the receiver declines your message because "most systems do include it, and it's lack of presence is highly correlated with spam email", then it's on the sender

Admittedly, the end result is the same.

I think it's the latter. But, in either case, you're right in that you get the same result.

Now, let's assume that if it is the latter (it's spam related), and Google were to accept the message, but then internally bin the message, it would be worse. At least in this case, they are bouncing the message. Because of this, the sender is at least aware that the message wasn't delivered.

Also, the author was able to get their mail delivered to a personal gmail.com address. The issue was with a Google Workspace custom email domain. This further makes me think of this as a security/spam related issue. Google is clearly capable of processing the message without a Message-id, they are just refusing for business customers.

My takeaway is that I think that Google is doing the least-wrong thing. And by being explicit in how they are handling it, it at least made the debugging for the author possible.

Also note: in a quick reading of RFC5321 (SMTP), rejecting messages for "policy reasons" is an acceptable outcome. I'm not sure if it applies completely here. The author should probably also be taking into account RFC5321 (SMTP) instead of just 5322 (message format).

> Also, the author was able to get their mail delivered to a personal gmail.com address. The issue was with a Google Workspace custom email domain. This further makes me think of this as a security/spam related issue. Google is clearly capable of processing the message without a Message-id, they are just refusing for business customers.

That's the annoying part to me.

An email is an email. By applying different rules for rejection on different mailboxes, gmail creates a system where it's harder for would-be implementers to test compliance.

If tomorrow gmail creates a new type of mailbox, will there be a third set of rules to have your message delivered?

There are dozens of spam and security settings that admins can change in the Google Workspace console, presumably because different businesses have different requirements. So in practice, there's not just two sets of rules in gmail -- there's probably thousands or millions (however many combinations of settings are actually in use).

Other anti-spam implementations also punish the lack of Message-ID. There are tools online that highlight this as an issue.

This here is a trivial case of simply not testing deliverability at all.

In my experience, email is an unreliable way to communicate any time-bounded critical information. When I want to be sure an email was transmitted on either side, the only reliable way to ensure this is to use a distinct channel to validate reception and confirm content.

That is, when some hotline tell me that they just sent and email with the information, I ensure they hold the line until I got the actual email and checked it delivers the relevant information to fulfill the intended process. And when I want to make sure an email was received, I call the person and ask to check, waiting until confirmation.

It’s not that much SMTP/IMAP per se as the whole ecosystem. People can legitimately get fatigue of "is it in my junk directory", "it might be relayed but only after the overloaded spam/junk analyzer accept it", or whatever can go wrong in the MUA, MSA, MTA, MX, MDA chain. And of course people can simply lie, and pretend the email was sent/received when they couldn’t bother less with the actual delivery status.

There are of course many cases where emails is fantastic.

Email is an unreliable way to communicate any information, in the strictest sense of the word "reliable." The protocol does not guarantee that any email will be delivered, nor does it guarantee that failure will be detected. It's a good-faith effort. The bits could drop on the floor at any point and you might never know.

Does it even matter when in reality it's more likely that this is intentional anti-competitive behavior by Google?

They once made all emails from my very reputable small German email provider (a company that has existed and provided email services long before Google existed) go into a black whole - not bounce them back or anything like that, mind you, their servers accepted them and made them disappear forever. I was in contact with the technicians then to get the problem fixed and they told me it's very difficult for them to even reach anyone at Google. It took them several days to get the problem fixed.

Of course, no one will ever be able to prove an intention behind these kind of "technical glitches." Nothing of significance ever happened when Google had large optics fiber connections with NSA installed illegally and claimed to have no knowledge of it, so certainly nothing will happen when small issues with interoperability occur and drive more people to Gmail.

At scale, it's very hard to distinguish malicious intent from the simple consequence of being the largest operator in a space so any motion one makes makes waves.

For what it's worth: having seen some of how the sausage is made, Google isn't particularly interested in screwing over a small reputable German provider. But they also aren't particularly interested in supporting such a provider's desire to route messages to their users because the provider is small. At their scale, "I've forgotten how to count that low" is a real effect. And email, as a protocol, has been pretty busted for decades; it's not Google that made the protocol so open-ended and messy to implement in a way that providers will agree is correct.

> Nothing of significance ever happened when Google had large optics fiber connections with NSA installed illegally and claimed to have no knowledge of it

Nothing of significance outside Google. Inside, Google initiated a technical lift that turned their intranet into an untrusted-by-default ecosystem so that data was encrypted on the fiber (as well as between machines within a datacenter, to head off future compromised-employee attacks). That process took at least five years; I suppose there's a scenario where it was all smoke and mirrors, but being on the inside in the middle of the process, I watched several C-suite who are not particularly good actors be bloody pissed at the US government for putting itself into Google's "threat actor" box and making that much work for the system engineering teams.

Also, an engineer at Google then made an end-to-end email crypto plugin for Chrome, including a flag that was a nod-and-middle-finger to the information revealed in the Snowden documents. https://techcrunch.com/2014/06/04/nsa-mocking-easter-egg-fou...

Thanks for this reply full of interesting information! These kind of comments are what I like about HN.

What does “not particularly good actors” mean in this context? Actors as in theater? Or “Bad actors” plotting evil things?

Actors in a theater. I have no reason to believe they were pretending to be upset because the were actually spies and knew what was going on before Snowden's reveals came out. They were as surprised as everyone else; the NSA wiretapping was done off-prem in cable that was privately owned (but stretched hundreds of miles and was, therefore, practically undefendable against compromise).

Long time ago when I was managing ISP email relay and customers asked "Where is the message I've sent?" seeing in the logs message accepted by receiving SMTP server was the end of the debug for me: I just handed the customer the part of the log and suggested talking to the receiving side IT administrator.

On the other hand, by erroneously treating a SHOULD as a MUST, I would say that Google is the one who's not RFC-compliant

Google is rejecting it to ensure incoming messages aren't spam. SHOULD means "you should do this unless you have a really, really good reason not to." Do they have a good reason not to? It doesn't seem so, meaning Viva is in the wrong here.

No, SHOULD is defined in the RFC, not by colloquial usage. Google is on the wrong, regardless of their "safety" intent.

After all, linguistics is full with examples of words that are spelled the same, but have different meaning in different cultures. I'm glad the RFC spelled it out it for everyone.

The RFC says a SHOULD is to be treated like a MUST, but well-justified exceptions are allowed.

RFC speak requires you to think for a while about skipping a SHOULD. It doesn't require strong justification.

When producing a message, it SHOULD have the id. With or withot it is compliant.

On the other end, we may receive messages with or without. Both are valid. We MUST therefore accept both variations.

The second one is a consequence of the former. So yes Google is the violating party.

No it doesnt lmao. It's quoted all over this thread and clearly is not in any way like a MUST

if Google's choices are protecting users, they can't be in the wrong. That's the reality of a shared communications infrastructure regardless of what the docs say.

When the docs disagree with the reality of threat-actor behavior, reality has to win because reality can't be fooled.

Spam senders don’t have pseudorandom number generators?

They're more likely to put in the least amount of effort or care the least about the reasons how the header is used later on.

Did i miss the part of the RFC that says google must accept every message? Pretty sure the RFC allows email providers to reject any message they feel like.

RFC cannot force a mail server to accept spam. You may argue that requiring message-id is a bad anti-spam policy but it does reduce amount of spam. In my observations around a half or messages without message-id are spam. I would not use personally this as the only reason to reject a message but I understand why someone may choose to do.

The RFC says a SHOULD is to be treated like a MUST, but well-justified exceptions are allowed.

Per RFC2119: 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

So, it's fairly explicit that the sender should use message-id unless there's a good reason to not do so. The spec is quiet about the recipients behavior (unless there's another spec that calls it out).

Not a specification but "Be liberal in what you accept?" comes to mind. (which I always personally hated but i'm just one shoveler).

Postel's law was a precept of the Internet of the 80's and 90's, when due to the primitive software engineering practices at that time, implementations couldn't be tested properly. That lead to many cases of poor interoperability, and it's no longer a good idea: for example, when HTML 5 was designed, they decide to put into the spec how to deal with the frequent errors like mismatched closing tags, etc... because all major implementations were "liberal" in what they accepted, but each in a different way.

What is the point of SHOULD then?

(No seriously, I’m asking; are there examples of where it’s actually different from a MUST)?

Also this reminds me of something I read somewhere a long time ago: when specifying requirements don’t bother with SHOULD. Either you want it or you don’t. Because if it’s not a requirement, some people won’t implement it.

I guess the one time it’s good is if you want an optional feature or are moving towards requiring it. In this case Google has decided it’s not an optional feature.

Typically, MUST means that if you don't do that then something will break at the protocol level.

SHOULD means that if you don't that, bad things are likely to happen, but it will not immediately break at the protocol level and during discussion in the IETF some people thought there could be valid reasons to violate the SHOULD.

Typically, IETF standards track RFCs consider the immediate effects of the protocol but often do not consider operational reality very well.

Sometimes operational reality is that a MUST gets violated because the standard is just wrong. Sometimes a SHOULD becomes required, etc.

Certainly for email, there is a lot you have to do to get your email accepted that is not spelled out in the RFCs.

SHOULD generally means: some people might require it. implement it for best results

backward compatibility makes it hard to add MUST. using SHOULD is a good alternative

"SHOULD generally means: some people might require it."

No it absolutely does not mean that. It means, by explicit definition which is right here, that text is exactly that definition, that no one requires it. They can't require it, and still be conforming to the spec or rfc. That's the entire point of that text is to define that and remove all ambiguity about it.

It's not required by anyone.

The reason it's there at all, and has "should" is that it's useful and helpful and good to include, all else being equal.

But by the very definition itself, no people require it. No people are allowed to require it.

Any that do, are simply violating the spec.

the spec doesn't include an obligation to deliver your mail. it parses fine, and is rejected by the larger system

> the full implications must be understood and carefully weighed

MUST means omission is unacceptable. SHOULD means MUST unless you have a good, well-reasoned excuse.

Incorrect. Not required is not required. You do not need to supply rationale or get agreement by anyone else that your reasons are good in their opinion and not just in your opinion.

Should just means the thing is preferred. It's something that is good and useful and helpful to do.

That is not "must unless you can convince me that you should be excused".

RFC2119

“ SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.”

I don't know if you thought you were validating or invalidating my argument, but this says exactly what I said.

Further, not only the presense of the words "should" and the doubling down with "recommended", also the absense of the words "must" or "required" or "shall" is pointed.

Because those words are used elsewhere. When something is actually required, those words are used, and so when they are not used, it is meaningful. It means no one is free to put their own words into the authors mouths and say "well they didn't say it but obviously they pretty much meant required" when they so pointedly chose not to say required.

And it's not a technicality. A thing either is or is not required. There are no other states. The spec does not require it, and so for Google to require it, Google is not adhering to the spec.

Since the spec does not require it, you can't say either that it falls under "full implications". "full implications" cannot include "Some servers might require it", because that would be circular logic. No server can require it in the first place.

I SHOULD have 8 hours of sleep every night. It's RECOMMENDED. However, there are times where it's best I don't (e.g. because of work, or travel, or needing to take someone to the hospital, etc.). It's definitely not that I MUST sleep 8 hours every night.

Yeah, and sometimes if you pull an all nighter bad things happen as a result of lack of sleep. So it is an apt metaphor.

“When jump getting over a wall, you SHOULD use three points of contact.”

For most cases you should use three points of contact. However, there may be other situations for example if someone is giving you a leg up, or you can pole vault, where another solution is preferred.

You assume that internet standards are prescriptivist; that the document describes how it is to be implemented. In practice it's often descriptivist, with the standards documents playing catch-up with how things are actually going in practice.

Anyway, in general you can expect that doing unusual but technically valid things with email headers will very often get your messages rejected or filtered as spam.

Standards are definitely prescriptive. But just like a medical prescription, it doesn’t ensure that actors in the wild will conform to what’s prescribed. People will not follow prescriptions for whatever reason, willingly or otherwise. It doesn’t mean the document wasn’t prescriptive.

For producers, ignoring a SHOULD is riskier because it shifts the burden to every consumer.

For consumers, ignoring a SHOULD mostly affects their own robustness.

But here Google seems to understand it as a MUST... maybe the scale of spam is enough to justify it. Users are stuck between two parties that expect the other to behave.

> maybe the scale of spam is enough to justify it.

This is 100 percent the case, and why these things are this way.

If you wanted to make email two point oh, I dont think it would look a lot like what we have today.

> This is 100 percent the case, and why these things are this way.

But gmail accepts emails without message-id on personal mailboxes apparently.

I think a mail 2.0 would be notify and pull based.... you notify a recipient's mail server that there's a message from <address> for them, then that server connects to the MX of record for the domain of <address> and retrieves <message-id> message.

Would this make mass emails and spam harder, absolutely. Would it be a huge burden for actual communications with people, not so much. From there actual white/black listing processes would work all that much better.

Is the idea that you could decide from the envelope whether you want to even bother fetching the message? Besides that I'm not sure I see the advantage

You have to have a working mail server attached to a domain to be able to send mail... that's the big part. Right now, email can more or less come to anywhere from anywhere as anyone. There are extensions for signing connections, tls, etc... but in general SMTP at it's core is pretty open and there have been efforts to close this.

It would simply close the loop and push the burden of the messages onto the sender's system mostly.

And yes, you can decide from the envelope, and a higher chance of envelope validity.

Like it proves you have the ability to receive mail at the domain you're sending from? I feel like SPF/DKIM already does this

SPF doesn't prove that... I can send through SendGrid with an SPF record, doesn't mean I've got a server configured to receive mail... for that matter, it actually makes it so you HAVE to be responsible for the mail system and cannot outsource sending separately from receiving. Again, shifting the burden enough to where other measures of dealing with bad actors are more effective.

https://jmap.io

jmap is the communication between a mail client and shared directory/mail services on a server. It does not include server to server communications (that I am aware of) for sending mail to other users/servers.

Couldn’t resist replying to:

> If you wanted to make email two point oh, I dont think it would look a lot like what we have today.

Google interpreted it that way because it drives more people to use gmail.

Reply on news.ycombinator.com