Hacker News Reader

Comment by cardanome | Hacker News Reader

Oh god, please just use primitive types. Don't make assumptions about things.

Everyone thinks they are so smart validating emails, phone numbers, zip codes and all until their great design goes live and they discover that users in the real world do not follow their assumptions.

I have seen that happen again and again. No, if your idea of validating an email is more complicated than "should have an @ symbol", I guarantee you, there is counter example that will mess you pretty system up. Have fun scrambling to fix that ticket.

Oh you think you know how an address and zip code should look like? No, you don't.

Please people, just use strings and call it a day. Why do you like to suffer?

I mean, sure, using the type system to protect you from mixing up units can be useful. Everything in moderation. Primitive types are good types.

View on HN

That sounds like an argument against excess validation rather than against types (maybe because of blog-driven development showing minimal examples of the power of types?).

Picking on one of those points, suppose emails are just strings. What then prevents them from getting used as names, identifiers, and other unrelated data? Just your continued vigilance as a programmer and hoping that nobody ever carelessly names the field "address" so that mistakes can slip by over a series of devolution commits.

You might not know much about what an email is, but you know _something_ about how you expect them to behave and be used, and if you like the computer to automate your work it's not totally unreasonable to rebind the string type as an email type and require explicit conversions at the point of use to treat it as anything other than just an email. There's zero runtime cost, it's not much more code, and that class of bugs is greatly reduced except for at the boundaries of the system.

Maybe that effort isn't worth it, or maybe your domain changes fast enough you'd have a lot of churn, or maybe those bugs aren't too important, or whatever. Using types to help you reason about the things you do know and do care about can be a huge productivity boost though, so I wouldn't just write the whole technique off.

> That sounds like an argument against excess validation rather than against types (maybe because of blog-driven development showing minimal examples of the power of types?).

A lot of developers confuse types and validation.

What? Would you rather have JSON as a string or as something like Map<string, JsonValue>?

> No, if your idea of validating an email is more complicated than "should have an @ symbol"

Then how is just using a string any better? Would you rather litter the entire codebase with validations that this is indeed a valid email or just do it once at the entry point?

I think excessive type system hacks are bad too but they are more often than not the problem of the language where its unable to express certain concepts naturally (see C++ template hacks).

> What? Would you rather have JSON as a string or as something like Map<string, JsonValue>?

This question can't be answered without additional context. What are the requirements?

But as a heads up, Map<string, JsonValue> couldn't be representation of all valid JSON documents. Lists are valid Json as well AFAIK. So I think this is making a point.

Based on the casing JsonValue is an object, not a primitive, so there's no reason that interface wouldn't work. It would just by a different subtype depending on what type the value is, and lists/dicts would also implement the Map<string, JsonValue> interface.

As much as I can just happily ignore JSON because I don't do webdev, I think there is a strong case to make that lists should be implemented using a native list, array, or vector class. But I won't argue because it's not necessary. The possibility for argument already proves my point.

My point is to show that "the type system" provides endless rabbit holes, and often is just a waste of time. There are a lot of situations where you want to represent a JSON document as a string. (Trivial example, as an embedding in an HTTP response object).

> The possibility for argument already proves my point.

Was just correcting something wrong:

> But as a heads up, Map<string, JsonValue> couldn't be representation of all valid JSON documents.

So,

> As much as I can just happily ignore JSON because I don't do webdev

I think this is why you're going wrong here. JSON is pretty much a solved problem, and Map<string, JsonValue> is pretty close to how it's used in Java and other languages that don't have native types that match its structure (like javascript and python do).

> (Trivial example, as an embedding in an HTTP response object)

This isn't json, it's what you get when you stringify/dumps/convert the json into a different datatype. When you need to be specific it's often called a "json string".

JSON is often interpreted as dictionary but is serialized as key-value pairs, for JSON itself this is not a big problem as everybody agrees not to produce JSON documents like {"a":1,"b":2,"a":3} so most people do not care about how their parser reads them.

In cases like URL queries or HTTP headers it is not such a clear cut. There it is common both to use duplicated keys and to use JSON-like dictionaries to read them.

Personally I never had bugs due to this: PHP does the "right" thing with duplicated keys and I never encountered it in node, but it bugs me that we use this kind of lossy[0] representations.

[0] in JS in particular the object are not really adequate to be used as dictionaries, inheritance and predefined keys aside there are also special magical attributes that behave in special ways Object.getPrototypeOf(Object.assign({}, JSON.parse('{"__proto__":null}')))===null;

JSON is a data-interchange format, i.e. a specification how to serialize and deserialize a domain of values. So Map<string, JsonValue> "isn't JSON" either.

JSON isn't a solved problem, it's a solution to a problem (and often used as a non-solution to a non-problem).

But none of that was my point.

> Would you rather litter the entire codebase with validations that this is indeed a valid email

Why would you? Just use it. For most of parts of the program it's not relevant to the computation whether the string is a "valid email", whatever that means. It's a string.

"Just use it" is what lead to the big SQL injection fallout and even today we pay the price as not even a year ago thousands of crucial service were vulnerable via log4j because of the "Just Use It" mantra.

You don't understand what I said.

I say, don't make assumptions unless you need them. Formatting an email address in an HTML document would work by wrapping it in the appropriate tag. (which implies html-quoting it correctly, but that is unrelated to email syntax).

Sending an email using an API would work by passing the email as a string to the API.

Looking up an email address from an address book using a pattern to match would be implemented with normal text search.

It doesn't matter if the email is "valid" or not. Don't overthink it.

SQL injection, XSS, and similar attacks are about incorrectly encoding embedded fragments.

The solution is either not to embed (SQL parameter bindings) or to always escape embedded fragments based on the embedding context

For reference: The Last XSS Defense Talk - Jim Manico - NDC Porto 2022 | https://youtu.be/wRC7jyhTkEM

You don't avoid injection attacks by validation, but by escaping.

escaping requires validation. You don't know what to escape if you aren't allowed to validate.

You don't know what you're talking about. For example, HTML escaping (a.k.a quoting) rules don't care what you're escaping - an email, a street name. It's just text.

And that's the point of it. You quote precisely because the container syntax doesn't know the syntax of what you're embedding. If it knew, there would be no need of the escaping.

It's called abstraction.

HTML escaping requires you to look for characters to escape for it to not interfere with HTML. This is quite literally validating symbols in the input. If they fail validation they need to be escaped. It's literally IMPOSSIBLE to do escaping without first validating every single symbol in the input.

So you're "validating" symbols now (strange choice of word). But certainly validating that the higher-level construct that we're escaping is an email address. Because the escaping procedure doesn't care.

Validating is just the process of ensuring an input is admissible in the way you want to use it. That can be symbols in a string, whether a string is an e-mail or even if a number is in a certain range. Escaping is just validation + fixup which can be used in some cases. Anyway the only way to validate an e-mail in practice is to use it and confirm.

> Escaping is just validation + fixup

You're confused and annoyingly persistent.

Escaping (or quoting in general) is a simple translation from a literal representation of a string to a (lexical) syntax representation with the purpose of embedding the string in an external medium (e.g. source code written in that lexical syntax).

Escaping is a mechanical process that doesn't discriminate between "valid" and "invalid". It is completely ignorant to the higher-level meaning of the string that is translated (e.g. email address) but solely operates on the constituent characters.

That is in contrast to validation, which is a simple function that decides whether a given object is admissible or not (as you say yourself). "Admissible" here is in with respect to a meaning that is higher-level than lexical syntax. It is semantic (is this a valid email), not syntactic.

(There are sometimes certain technical restrictions on which values can be represented in a lexical syntax, for example hard limits on string lengths. So there is a small extent to which "validation" can fill a purpose with relation to lexical syntax, too - but that's not what we're discussing).

To make it even more confusing, email addresses conform to a (albeit poorly specified) lexical syntax, too. And you can certainly attempt to validate if a given string is valid email address. However, HTML syntax doesn't care about that. Email address syntax is not part of the HTML syntax. HTML specifies how to escape _strings_, not email addresses.

And HTML syntax is right not caring about email syntax because it would be unnecessary complication in practice.

Just as the other examples I gave. E.g. looking up email addresses from an address book is not a task that in practice needs to be more specific than looking up a string from a list of strings.

> Anyway the only way to validate an e-mail in practice is to use it and confirm.

Which was my initial statement "Just use it" that you heavily disagreed with.

> Escaping is a mechanical process that doesn't discriminate between "valid" and "invalid". It is completely ignorant to the higher-level meaning of the string that is translated (e.g. email address) but solely operates on the constituent characters.

It does discriminate between "valid" and "invalid". This symbol is "valid" and we don't need to do anything. This symbol is "invalid" and we need to escape it. Validation occurs throughout the whole abstraction stack. Not only at the level of meaning of an entire string.

> Which was my initial statement "Just use it" that you heavily disagreed with.

In the case of e-mail I don't disagree with you. It is however balls to the wall insane to say "Just use it" in general. Which was my point. Notice how my reply specifically mentions vulnerabilities that were caused by the "just use it" mantra.

> This symbol is "valid" and we don't need to do anything. This symbol is "invalid" and we need to escape it.

It's quite a stretch to call symbols that need to be escaped “invalid”. And it's often possible to escape without discerning between “valid” and “invalid” characters. For example, in HTML you might just convert all characters into numeric entities.

> It is however balls to the wall insane to say "Just use it" in general.

Good thing the parent didn't say it in general.

I meant to say "certainly _not_ validating"

You seem to have a weird definition of “validation”.

My favorite "easy win" find from learning Haskell last year was just the `newtype` keyword, which basically just let you alias primitive types with zero runtime impact.

`newtype Email = Email String` and `newtype Username = Username String` are just `String`s with guard rails.

Python's type annotation system has this feature too:

    from typing import NewType

    UserId = NewType('UserId', str)

    user_id: UserId

    user_id = "abc"          # Type check error
    user_id = UserId("abc")  # OK

    # At runtime, a NewType is the identity function
    s1: str = "abc"
    s2: UserId = UserId(s1)
    assert s1 is s2

Docs: https://docs.python.org/3/library/typing.html#newtype

Runtime: https://tio.run/##TU49D4JADN37K544CIkLupk4OBoTXXQ2CKdcDHeXts...

Type checking: https://mypy-play.net/?mypy=latest&python=3.10&flags=show-er...

You probably want smart constructors, as you probably don't want the possibility for a 10,000 character username to float through your system.

The point is not about validation, it's about conveying semantic information through types. It's perfectly valid to have an email type that is just a wrapper around a string. The advantage is now you and all your functions unambiguously know that the type represents an email (whatever that means) and not a frobinator.

Thank you, I feel like I am going insane here.

About 90% of stuff I ever worked does email things here or there, and I never had the desire for some hyper complex abstraction with 10 pages of documentation just to store email. I also never experienced a bug that would have been fixed by this.

I mean if the user is putting in some bullshit there, the basic validation of type="email" or pendant if I am not working in HTML is gonna tell him that and after that he does or does not get his verification email. Boom, problem solved along with other one's.

Need just the domain? Well I'll use a damn 1 liner function with split() or something for this hyper specific use case, don't encumber me with the probably faulty abstraction of email some library designer cooked up 8 years ago just for this?!

I do use types when they are actually convenient, but these types-are-the-best-everywhere-all-the-time articles keep failing to convince me...

Just because there are cases where over-validation (email address being the primary example) can be a problem doesn't mean that _no_ type validation is useful. There are many, many places where validation-beyond-primitives is useful, or even just types that need actions taken on them before they can be used in place of another type. One simple one is "non-negative integers", not commonly provided as a primitive type but _very_ common to need to be able enforce. Complex numbers are another. Normal strings vs HTML strings (need to be properly handled before they can be output to a front end). The list is endless, and custom types can provide a _lot_ of safety.

There's a sharp distinction between validation and typing. I can cast a string into a domain-specific Email type without validating the string. I can also reject a string due to a validation rule without changing its type.

Opaque type aliases are great because they impose semantics. Even if I don't know how to validate an email, it's nice, at times, to distinguish a string which I suspect to be an email from all other sorts of strings.

Using specialized types can be useful, but if it provide s methods they need to be correct for ALL situations.

Here's a subversion of GitHub's authentication (now fixed) where they assumed that "lowercasing domain name using English case rules is always fine and produces the same result" led to a vulnerability: https://dev.to/jagracey/hacking-github-s-auth-with-unicode-s...

TIL that punycode breaks email.split('@') to get the domain:

Apparently John@Gıthub.com normalizes to xn--john@gthub-2ub.com but Gıthub.com normalizes to xn--gthub-n4a.com.

For now I will go back to forgetting that emails not covered by /[A-Za-z0-9.-+_]+@[A-Za-z0-9.-_](.[A-Za-z0-9.-_])*/ exist to preserve my sanity but this does confuse me.

People are getting lost about your example as they don't see the analogy as it sounds like validation of input. The point is still there if people are willing to look past the confusion, that making a type hierarchy requires you clearly know upfront what different subsets of the data you deal with actually look like and this will change at different stages of the project. This is the reason people ditched c++ and java in 2000 and started hacking in python and js, at least at the beginning although things are shifting.

The thing is a lot of things are reversed here, the reality is the way evolution is more sensible is to "generalize" from the bottom up, unless you are very sure from the start that certain types categories make sense for your problem. That's how things should always be, you can only generalize once you've worked through a problem and realized certain aspects are actually common in someway and can be connected somehow, and thus you can take a subset of data you have in your program and organize it into a type. The thing is I feel like that is even harder than thinking top-down (which I feel like is where the pendulum is swinging now) because that takes time.

The reason top-downers still do their thing is they feel their method is better is selective thinking (sorry, I know this is harsh) because they honestly discount the anger and fury people from the outside have dealing with their code and they ignore the amount of refactoring they need to do when their designs break. In fact, they love the churn really or at least merely accept it as a "part of development."

To me, type is for data abstraction, like we use generic for function abstraction.

Abstraction in this case, mean, for future changes, i just need to change in one place, the Email type abstraction instead of searching and replacing every ussage of primitive email string.

Ahem. Not all email addresses have an @ in them. If you're sending an email to another user on the same machine, just their username is enough (at least for some mail implementations).

Note that this makes your overall point stronger, not weaker.

Unless you're really going to use textareas with scrollbars for every string and store everything in unlimited length database fields, you probably do want to distinguish between single-line and multiline strings and have some limit on their lengths.

> No, if your idea of validating an email is more complicated than "should have an @ symbol", I guarantee you, there is counter example that will mess you pretty system up.

Requiring b2b users to use their @company.com email is common and some b2b customers actually expect to be able to configure that. Another simple case is stripping out any "+whatever" from gmail addresses and to flag that sort of thing for other eomains so support can verify it's not a case of someone creating 72 trial accounts. Yes, rejecting users due to naive and incorrect validation is bad, but treating emails as entirely opaque strings isn't always an option either.

> Another simple case is stripping out any "+whatever" from gmail addresses and to flag that sort of thing for other eomains so support can verify it's not a case of someone creating 72 trial accounts.

trial1@mypersonaldomain

trial2@mypersonaldomain

trial3@mypersonaldomain

But for that you need a personal domain and the ability to configure something like that, whereas everyone in a corporation that runs their email through Google can use the +abc thing out of the box, and I've seen it used a ton. The goal with measures like that is usually not to catch every single one who tries to game the free tier but to ensure it's more inconvenient to do so than just buy it for most people, and typically filtering +abc emails will be just one of several different measures taken to that end.

There are sites where you can easily get all addresses of the format something@yourname.sitename.xxx.

> if your idea of validating an email is more complicated than "should have an @ symbol", I guarantee you, there is counter example that will mess you pretty system up.

unless you are writing an email server. Then you would need to do this properly.

In other words, validate the data that is in the domain of your application. If your app simply _sends_ email, then it's not your domain, and don't need to validate, as long as the receiving end of the email (aka, the email server) accepts it.

You are talking about validation not types.

Reply on news.ycombinator.com