Hacker News Reader

Comment by dan-robertson | Hacker News Reader

I think avoiding numeric types is a good decision. It tends to eventually cause problems when one implementation converts numbers to doubles, another to either doubles or longs depending on whether they have a . or e, and another which converts them to bignums (or passes them as strings to the caller).

One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.

There are also issues when something looks numeric but doesn’t parse (eg 1.2.3, 3/2, 12in, 4h30m2s, 2:30, 2020-02-29, etc). One way to deal with these is a tokenisation rule like in Common Lisp: if it is a valid number syntax then treat it as a number, otherwise it’s a symbol, but this can lead to issues (eg you would need to know that when your number needs more than float precision or otherwise doesn’t follow the rules, it should be in quotes. It seems crazy to pass that detail on to the poor sod who has to write the config file).

View on HN

> One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.

The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way.

Getting rid of yaml's 30 ways to write "true" and "false" by making everything is a string just means that you now have 30 tool-specific ways to write "true" and "false".

The "everything is a string" approach already exists in shell scripts and TCL and it's not really that great.

Except YAML is straight-up wrong at times, unless you know all the edge cases. I just learned unquoted NO is coerced to False. A classic case of leaky abstraction/"bad magic".

Unless you have actual type annotations/tags (eg xml, jsonld, graphql), everything IS a string. There's no assumptions otherwise.

https://yaml.org/spec/1.2/spec.html#id2805071

Of course, even though YAML 1.2 is a decade old, there are still many parsers that accept YAML 1.1.

Huh, I had no idea, thanks for the heads up!

That seems like a pretty sensible approach - eliminates the "Norway problem" altogether. I'd guess you have to use the "%YAML 1.2" directive in all documents in order to get it though...

You don’t need this directive to get YAML 1.2 parsers to parse your document as YAML 1.2; that’s the default. You only need it to instruct YAML 1.1 parsers to raise a warning (and I guess for parsers of future version of YAML to fall back to 1.2 behavior, if that ever happens).

https://yaml.org/spec/1.2/spec.html#id2781553

https://yaml.org/spec/1.1/#id895631

> Unless you have actual type annotations/tags (eg xml, jsonld, graphql),

YAML has actual type annotations (tags).

Wait, really? scrambles to check woah...

I've never seen typed yaml, this is wild.

    negative: !!int -12
    zero: !!int 0
    positive: !!int 34

Can't say I love the notation, but indeed that is type annotations. I guess neither "yaml type hints" nor "yaml type annotations" are the right query. Had to search explicitly for "yaml tags".

The neat thing is that you can use tags to represent custom data structures or functions. For example, AWS Cloudformation uses YAML tags as a shorthand for template functions [0].

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...

Depending on the parser you can also use them to call arbitrary code. This used to be the default behavior of `pyyaml.load`.

Types are !!screaming at reader. Don't want to start blameshitting, but is yaml supposed to be a simple human-readable format?

It is screaming with double negations!

It's not screaming without double negations!!

I would hazard to say that any magic is bad in engineering. (Sorry Perl fans!)

There isn’t really any magic in Perl, just lots of unfamiliar lexicon if you come from more traditional programming languages. Perls problem is it’s history is rooted in command line usage so there’s a tonne of inherited reserved single character variable names and such like that are optimised for keystrokes rather than readability. However you can certainly write Perl programs that don’t follow those older conventions and look more like a modern language.

> There isn’t really any magic in Perl

http://p3rl.org/guts#Magic-Variables

    perl -MDevel::Peek=Dump -mTie::Scalar -e'
        //g; Dump $_; tie $c => "Tie::StdScalar"; Dump $c; Dump \%ENV
    ' 2>&1 | grep MAGIC

That kind of magic in Perl has a different kind of meaning than "magic number/bool/string parsing" in YAML.

Perl tied-variable magic just means there are (effectively) getter and setter properties attached to the variable. "Magic" is just the name that was chosen in the implementation, and it stuck.

It's used to implement variables with special, automatic meanings, like $$ for "current pid" and $! for "last error".

It's also used to implement variables with user-defined behaviours on access, which is quite handy for a lot of abstractions.

A lot of modern languages support both of these things, because they are useful, but it's not called magic in those languages, it's called something like "watchers", "proxies", "getters and setters" or "hooks".

No, the criticism of YAML-style "magic" is that it leads to entirely surprising behaviour from innocuous input. Perl magic is not that kind. If you're using a special variable, you already know why.

That’s just playing around with the same reserved variables I spoke of before and there is nothing magical about them aside their silly name. In fact the opposite is true, they’re actually predictable and well documented. They just happen to have terse names as a throwback to command line usage (eg you wouldn’t call $1 a magic variable in Bash because it happens to work the same as ARGV[1]).

In fact in Perl, you can opt for longer, readable, lexicon over the terse single character variables; and that’s literally how modern Perl should be written.

Whereas the problems described with YAML is where it can automatically alter your data based on what the parser “thinks” the data should represent. Which is generally what people mean when they talk about “magic” in IT: systems that don’t honour your input and instead automatically convert it into something else. Perl doesn’t do this even in spite of it looking like executable line noise to many.

I think "magic" is often just abstraction. And while abstraction is certainly necessary to speed up work and declutter the brains of the end users, too much abstraction takes control away and bad abstraction takes the wrong decisions in your name.

If you take a look at how string handling works in most programming languages there is a lot of "magic" going on there. Which isn't a bad thing necessarily, because most programmers don't want to deal with the intricates of strings unless they really have to. The key is that this magic doesn't get in your way and doesn't do too magical things nobody ever asked of it.

technology without magic? nonsense

Insert Arthur C. Clarke quote here.

That was the GPs point, YAML contains a lot of insanity, but let’s not throw out what it got right along with everything it got wrong.

Encoding meaning in whitespace is an abomination. That’s all I have to say about YAML.

Encodingmeaninginqhitespaceisanabomination.That’sallIhavetosay.

Encoding meaning in whitespace is an abomination. That’s all I have to say about Python.

> The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way.

In Python:

    >>> import json
    >>> json.loads('{"x": 9007199254740993}')
    {'x': 9007199254740993}

In my browser's JavaScript console:

    > JSON.parse('{"x": 9007199254740993}')
    {x: 9007199254740992}

(Consider what happens if you try to send a tweet's ID, a perfectly normal number like 205052027259195393, through JSON. Or if you try to serialize a stack trace on a 64-bit system, where addresses are also perfectly normal numbers.)

I see what you are getting at and it is worrisome indeed, however, why would one encode an id as simple number?

Why wouldn't you? It's a number. It's also sorted (up to about one second of inconsistency between Twitter worker processes on different machines), and you want to sort them by numeric sort, not by lexicographic sort as you would with strings.

I mean, there's a pretty clear argument here: Twitter themselves used to return these numbers as numbers in the API until they realized they were about to hit this problem. https://developer.twitter.com/en/docs/twitter-ids

Using numeric types as ids probably leaks into other areas, where being numeric is then assumed. When you want to change it at some point, you got to be very careful suddenly. A string could simply contain a new kind of id. Seems less refactoring effort would be required. On the other hand, switching from numeric to strings might give you compiler errors when types do not match, so perhaps it might even make refactoring simpler.

You can always add a second or third key. Stop messing with the primary key. Many systems have a numeric primary key and then only expose a hash or UUID to the public.

>why would one encode an id as simple number?

That's an incredibly weird complaint when the real problem is that javascript's JSON.parse doesn't use BigInt for large numbers.

I have very little understanding of JSON, so why would this happen?

It's more a JS thing: The lone JS number type is double-precision floating point, meaning beyond a certain range there are integers that cannot be accurately represented as a JS number.

The JSON standard doesn't place restrictions on size or precision of numbers, instead just noting that implementations can vary their treatment of and limits on numbers. While JS uses doubles for all numbers, many other languages emit an integer type for a JSON integer. So, once you go beyond the range where a double can accurately represent all integers, you run the risk of a mismatch in how the number is interpreted by different languages parsing the same JSON.

Of course the spec also allows you to create way too big or too precise numbers that would be problematic in most languages as well; it's just that this is a somewhat common bugbear.

I wouldn't necessarily call it a flaw in JSON though, more an issue with JSON.parse or really just a fact of life when dealing with numbers in JS. Alternatives to the built in JSON.parse exist to read large integers as strings or bigints.

JavaScript is using double as a number. There's no such thing like integer or floating point numbers per se, only so called Number.

Described issue is not problem of a JSON but engine which parsed it and language which stands behind the parser. Any config format will eventually have same result and same issue.

So forcing programmer to parse every single piece of data for sake of "it's his responsibility" is not a case here.

I also disagree this is in any way programmer responsibility to create standardized way of creating parser for everything. This format gives you nothing but indentation so you are forced to create documentation for everything field, what type it's and what kind of values it takes. Lots of extra work for nothing when you have any other format.

It's a Javascript problem. Integer caps out at 9007199254740991 (253-1), and after that it's treated as a BigInt. As long as you're working with BigInt literals, it'll be accurate, but when you convert back and forth you can lose precision.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Python promote to Bignum automatically.

Javascript number is a floating point

    Number.MAX_SAFE_INTEGER
    // 9007199254740991
    Number.MAX_SAFE_INTEGER + 1
    // 9007199254740992
    Number.MAX_SAFE_INTEGER + 2
    // 9007199254740992

no automatic promotion to BigInt

    9007199254740991n + 2n
    // 9007199254740993n

The example uses numbers outside the range of JS numbers (too many significant digits), so when the JSON is evaluated as a JS number some of the least significant bits are rounded. Personally, I think it is a mistake to use values in JSON that can’t be represented in JS, but the standard doesn’t explicitly forbid it.

JavaScript numbers are double-precision floating point numbers.

The solution to such problems is a better standard for the handling of numbers, not dropping support for numbers altogether.

> The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way.

I don't think that's what this tool is for. This tool is for humans to read and edit data. That's a different use case from automated programs exchanging data, for which I agree you should be using standardized numeric and boolean data types and not making everything a string. But how that standardized data gets determined from data that humans enter should be up to the individual application.

Agreed. Serialization is not really possible with this tool, due to the following:

> A key that requires quoting must not contain both single and double quote characters.

You can't really serialize user data with that restriction.

It is still useful for humans to have a well-understood way to input typed data.

If you make everything a string then the interpretation of "no" as boolean true or false is left to each tool, and there are even tools which have different interpretations of "yes/no" for each field.

What is the use of this then? Humans don’t need structured data to exchange information and if this is insufficient to communicate it to the machine then it won’t be a great human to machine format either.

> Humans don’t need structured data to exchange information

Maybe we don't need it, but it often helps.

Also, this data format isn't necessarily for humans to exchange data with other humans, but for humans to give data to applications in a format that's much easier for humans to use.

> if this is insufficient to communicate it to the machine

Not at all. Each specific application can easily parse this data format according to its own needs. What this data format doesn't specify is a single translation into application data that is the same for all applications. But applications don't need or want that, because they have different use cases.

I see. That makes sense. Now that I think about it maybe if its in front of some database or storage that only handles strings it would work well as well.

I think this is an “input format” vs “serialization format” issue. NestedText is meant for user input so it’s focusing more on the UX of the syntax. In that case, having to parse strings is a reasonable task for developers to take on

If NestedText is meant to be written by non-programmer humans, the details of null, undefined, 0, false, "NO", nil, or whatever else, would likely also be lost on them.

Most likely if their input is meant to be machine interpreted they would need to be trained to provide specific inputs anyway. I like that NestedText doesn't hide that problem. It lets the user organization decide how it wants to manage that problem, and what symbols or words are understood by the people authoring the files.

>you now have 30 tool-specific ways to write "true" and "false"

ISO8601 joins the chat.

> I think avoiding numeric types is a good decision.

Only if this format is intended for use-cases that never need to deal with numbers.

That statement couldn't possibly be more wrong.

Number parsing (and encoding!) is a decidedly non-trivial problem. You need to concern yourself with -- at a minimum -- all of the following:

- Unsigned 64-bit numbers.

- A series of digits that would be bigger than a 64 bit whole number. Convert to float? Truncate in some way? Error?

- NaN

- Infinity

- Negative zero

- Denormal numbers.

- Differentiating between decimal/currency types and floating point numbers. Not all decimal values can be exactly represented as floats!

- Efficiently encoding floating point to use the minimum digits without losing precision.

- Parsing those minimal numbers with perfect "round-tripping".

- Doing the above efficiently.

- Securely too! Efficient parsers cut corners on sanity checks. I hoped you fuzzed your parser...

The above can easily amount to many kilobytes of extremely complex code. Look up "ryu" as an example of what Google came up with to make JSON number parsing reasonably efficient.

Meanwhile, reading a fixed-length number from a binary format can be done in a single machine instruction. One. It might not even take an entire CPU clock cycle! Okay, two, if you need to bounds-check your buffer, but there's ways to avoid that.

Afterwards, the bounds check is again literally just two machine instructions in complexity. That's not the difficult bit!

The difficult bit is the parsing.

You’ve given lots of examples of things that make parsing numbers difficult but I don’t see why they are relevant to a config file written by humans. I think it makes sense to have the number parsing owned by the thing which cares about the number format.

One example you provide is decimals for currency values but I claim you would want such values to look like $1234 in config files so that when they are reviewed or written, the person reading the file knows they are looking at a dollar value and can be concerned if it is too large.

I’m not suggesting that applications write their own number parsing. Just do uint64::parse or parseInt or Double.of_string, or whatever else you need to access your language’s number parsing routines.

Is the format written and read by a human?

> Just do uint64::parse or parseInt or Double.of_string, or whatever else you need to access your language’s number parsing routines.

Okay, so the computer is doing the parsing.

Those functions are notoriously inconsistent in their behaviour, particularly across different programming languages. If you're not careful, you'll end up accidentally using the internationalised versions of those functions. Even if you're careful, other people won't be.

Remember, data formats are for interchange. They have to be language agnostic. They have to be well-defined, and it should be possible to write a parser for them without having to guess at the precise details.

The harmful consequences of the Robustness Principle are now well-recognised in computer science: https://tools.ietf.org/id/draft-thomson-postel-was-wrong-03....

Some things need to be done properly, nor not at all.

If you go fully against Robustness principle, you lose the reason to use textual formats as well, since they are designed to be forgiving of human errors in input and catch them in syntax.

And - it is certainly OK in many instances to have fixed-width, fixed byte-order binary encoding as the format's basis. It comes with the twin downsides of wholly different categories of errors cropping up, and with the lack of a universally agreed upon tool for human entry.

Perhaps text was a fashion, though. I definitely have had thoughts in that vein lately. And in that case we shouldn't always be rushing to use it as the source of truth when we have many good, machine-level agreements about numeric formats.

I wrote some config for my application, which knows how to read it. Why do I want some other application In some other programming language to read it too?

I am far more worried about localisation issues than language issues. If you are storing something central to multiple applications I'd argue a text file is the wrong tool

But which of these are problems in configuration files written by a human? That is the aim of the format. Moreover in applications were there could be issues, it would most certainly be tied to very specific fields and you would want specific application logic to handle that field. Now if people misuse it as a data exchange format or so, yes I agree with you, but at that point just use a binary format instead.

> That is the aim of the format.

That doesn't matter at all. The author's aims will be ignored if this format is used for anything even vaguely important. Eventually it'll need tooling to both read and write it.

DevOps pipelines, applications with GUIs, or something will need to both parse and generate this format in a consistent way.

There is no such thing as a human-write-only format in widespread use.

Even programming languages are regularly generated by tools such as RPC API codegen tools, LINQ-to-SQL and the like.

In my opinion, the best solution to these issues is to:

1. Declare numbers as numbers in the configuration language. E.g. "decimal(1e1000)".

2. Parse declared numbers with a lossless format like Python's decimal.Decimal.

3. Let users decide at their own risk if they want to convert to a lossy format like float.

Then you just roll your preferred serializer on top of format in a properly composable form.

I am of the opposite view: As long as there are no edge cases, supporting more types in data languages is good, it leads to failing faster.

Too many types could get overwhelming, but I like where amazon's Ion is [1]. It actually supports multiple number types, with decimal being the default for values with a dot.

> you would need to know that when your number needs more than float precision or otherwise doesn’t follow the rules, it should be in quotes

Not really. The configuration value should either be a number or not, which is determined by the application reading the config. As a config writer you only care to make the type match (so, in json, if the application uses number you make sure you use number, and if it expects a string you use that)

(disclaimer: I work for amazon, but have nothing to do with Ion other than having used it. Opinion is my own, not my employer's, yadda yadda)

[1] http://amzn.github.io/ion-docs/

I wonder if you could just include explicit type information?

  name(str): Dave
  age(int4): 22
  dob(date): 2020-02-01
  photo(base64):TWFuIGl....

That's not the only way to deal with them, you can specify type like bson.

The nice thing about that is it solves the problem rather than hoping it doesn't matter or assuming each program's validator will think to note all of the possible data types not compatible with the program natively. If a u32 is defined in the file and you've only got doubles to work with it's a given you'll have to deal with it in your tool specific validation. For everyone else it's well defined.

The downside is it's a bit more verbose and if you have all of that info already it's pretty easy to jump to just using a binary format which will be more efficient anyways.

My ideal config language would have unambigous syntax for at least 64-bit signed integers and doubles (so, for example the spec for the language requires 56 to be parsed as an integer and 56.0 to be a double. Additional types would be ok too, as long as the syntax is unambiguous and obvious.

Both of those are true for TOML.

The spec requires that a full 64 bits of signed integer be parsed and understood, that hexadecimal, octal, and binary values be integers, and that floating point values be parsed as doubles.

It doesn't support hexadecimal float, however, which is a pity: having a guaranteed bit-identical format is a nice affordance.

> I think avoiding numeric types is a good decision. It tends to eventually cause problems when one implementation (...)

It seems you're mixing up the language definition with implementations that try to follow the language definition.

If different implementations have different results then either they are buggy or the language has some important holes in the specification.

Either way,the solution to this problem is not less validation.

That's the whole point, isn't it?

I mean, if you already acknowledge the fact that this parsing and validation is a basic requirement, why handle it as an afterthought and force developers to add their own hand-rollef absurd and unnecessary type checks and type coversions?

Wouldn't it simply easier to let the language and the parser do that already?

I mean, no one ever complained that JSON had string types. In fact, one of JSON's main complaints is that it doesn't support enough types, such as timestamps.

Reply on news.ycombinator.com