Hacker News Reader

> Naive people find liberation in text-based protocols that don't require tools to understand.

I don't think text is necessarily the problem here. Other protocols might allow you to use less memory, but they aren't running into a memory limitation in this instance.

You can divide parsers into two groups: parsers designed to read data which follows a specific schema and parsers designed to read arbitrary data.

Schema-aware parsers have additional context which allows them to process the data much more efficiently. The stricter the schema, the more efficient the parser can be. Meanwhile, parsers for arbitrary data have no such context and must rely completely on syntax to parse the data. A poorly optimized schema-aware parser may still outperform a well optimized parser for arbitrary data.

You can still encode data as text and even use popular syntaxes like JSON or YAML with schema-aware parsers. I've personally helped design a schema-aware parser for XML data. But you should keep in mind that most third-party parsers for those syntaxes will be designed to parse arbitrary data. Parsers for arbitrary data can still be very fast - fast enough for many purposes. However, schema-aware parsers have a significant advantage when it comes to computational performance.

Text is the problem, here and everywhere else. Text encourages looking for delimiters, escaping, and flexibility (optional whitespace, case insensitivity, leading zeros, redundant signs, newline style compatibility). Compared to binary with length prefixes, that’s really slow, prone to injection vulnerabilities and subtle incompatibility, requires copying, introduces more side channels (the length of your message now depends on the values of numbers inside it!), ….

And once you get down to it, text still requires tools to parse. I routinely run stuff through `jq`, because despite JSON being a text format, actually consuming it as a human requires a tool. (Often to filter it, but sometimes just to colorize the output to make it legible.)

Text =/= Variable Length

I can store fixed-length data using text. A strict schema can specify things like fixed lengths, value padding, orders for maps, exact whitespace rules, etc which can entirely remove the need for delimiters. A schema could even require length prefixes, just like you might use for a non-text format.

These concepts do not necessitate abandoning text. If you need the data to be readable/writable by a human with minimal tooling, you can still leverage text without sacrificing much parsing efficiency.

In theory, sure. In practice, nobody does, and length prefixes stop a format like that from being conveniently writable with a plain text editor.

I work on avionics and we definitely use fixed-length text all the time. Text is often much more convenient than blobs, even when adhering to strict schemas.

Syntaxes like XML, JSON, YAML, etc have become popular for their convenience and versatility, but they do not represent the whole of what is possible with text.

You've not used enough Ada: its variable length string is so clunky that fixed sized text is used instead..

That's orthogonal. Msgpack is basically json but easier to parse from C, and less inefficient on floats and integers.

Naive people find liberation in text-based protocols that don't require tools to understand.

As a huge fan of text-based protocols (but not of JSON or YAML), calling a whole class of debuggability, interoperability, and onboarding benefits "naive" is a bit much.

It's a lot of the reason why software has gotten to be so porous in terms of security flaws.

I would trust programmers at Intel to handle FORTRAN-style data structures, and maybe even make a compiler in C.

I wouldn't trust anybody from Intel to write a web server or even a web application with PHP in that wild west of applications programming with strings they are sure to mess it up and thus leave a vulnerability in the management engine on your CPU. It might superficially look like good code, but American Fuzzy Lop will find the branch that isn't obvious, and so will some kid.

To me the problem seems to be old-school C programmers thinking that they don't need to use a library to parse a standard format. I have only ever seen JSON parsing errors in C and C++ codebases because they are the only languages where developers seem to think it's a good idea to parse JSON using string functions rather than a JSON parser.

Some of that is that they don't have systems like maven or npm that would make it really easy to incorporate libraries into the build.

That said, what I like about C is that it is a simple language where you can throw out the standard library and start anew: it makes me think of the old versions of ALGOL that didn't specify mechanisms for I/O, and now we know you can specify that behavior in libraries and leave the language pure.

> Some of that is that they don't have systems like maven or npm that would make it really easy to incorporate libraries into the build.

Definitely! I wish the C and C++ communities would take this problem seriously.

> That said, what I like about C is that it is a simple language where you can throw out the standard library and start anew

This definitely is a nice property. But it's not just C that can do this. You can for example do this with Rust, and get an excellent package management system (that can indeed package these alternatives to the stdlib) to boot.

As a C programmer, i can say that most of the json libraries fail on simple fuzzing test, or do some other idiotic stuff like trying to handle numbers (and doing it wrong of course). We have awesome parser generators such as ragel, yet people still handroll parsing code that is both buggy and slow.

> We have awesome parser generators such as ragel, yet people still handroll parsing code that is both buggy and slow.

Wasn't ragel responsible for the cloudbleed vulnerability? Wherever possible we really shouldn't be using C at all.

No, the bug was in code that was feeding input to ragel [1]. Ragel itself was fine. (Ragel also can output to non C languages). I've used ragel myself extensively and it's really rock solid, I've fuzzed the state machines it generates a lot. I've even used it to parse protobuf specifications and generate code from them, the company still uses that code. (and it's really not a lot of code)

1: https://blog.cloudflare.com/incident-report-on-memory-leak-c... (As for cloudflare, I don't think it's good idea to let single company to MITM the whole internet)

It sounds like Ragel still requires the user to "use it correctly", which is really not great.

That's like saying driving car into a wall full speed should not kill you

It's like saying that we ideally shouldn't have cars that can drive full speed into a wall, or at least that these cars shouldn't be the cars that most people use. And indeed modern cars will generally at least attempt to break before they hit a wall. It seems not too fanciful to think that in another 10 years we will have cars that can reliably prevent themselves from being driven into walls.

Our tech with regard to type and memory safety is more advanced than our car tech: we already have the tech to prevent these errors. And I believe we should be using it wherever possible. Obviously there will be exceptions (the more obvious being legacy codebases), but they should be exceptions.

My Subaru will do its best to turn off the gas and auto break when it detects I’m about to drive into a wall.

I used to be the only one where I worked who would use the standard library for xml in perl rather than just ad hoc regexes.

They'd recycle my scripts when dealing with very large files but wouldn't learn it themselves.

> variable-length data structures are much more treacherous than fixed-length data structures

A fixed-length structure is only fixed until you need to add a field to it. I understand the appeal, but good solutions seem to be very rare here. ASN.1 was possibly the first attempt but turned into a total nightmare. Protobuf seems to be doing a reasonable job, but only as a wire format and not a persistence format.

Text always "works", even if inadequately. And it can be handled with your normal diff and editing tools.

We've been using protobuf as a persistence format without issue?

A self-describing format like MessagePack can be easier to build readers for in other applications, or if the schema is lost or changed.

If we've lost access to the schema, then we've lost access to our entire source repository, so we probably don't need to read our data anymore?

I guess I could see that being an interesting property in software that wants to deal with incompatible forks, or software that for whatever reason wants other people to read their data but is unwilling to ship a schema with it, but just for persistence on backend servers it doesn't seem like a very interesting property or worth the extra space requirements.

CBOR is the gold standard for schema free data.

You have. Far more apps out there are using JSON or XML, including (as we've seen GTA).

Ah, you were just talking about popularity not fitness? Fair enough.

> Text always work

Windows vs Linux end-of-line markers often screws-up things. UTF often screws-up things. Tab-vs-space sometimes screws-up things. Text is related to Postel's principle, which is often questioned because it leads to sloppy implementations on one side and complex implementations on the other side, and other bug-is-now-a-feature niceties.

I designed a couple of small internal protocols, and have been maintaining them for at least a decade. Maintenance time is nearly 0 because I never had to add a field to them. Even though the software using them evolved a lot and in unexpected ways.

Yes, they stood the test of time because they solved the problem we had, and those problems didn't evolve much. Not every protocol or format needs to be hyper-extendable and scalable.

> even if inadequately

So it works poorly but it works, so it can be shipped. That's why you buy a new smartphone every two year and don't see much improvement in response time or battery life.

Whenever floating-point numbers converted to text are not intended to be used by a human, converting the numbers to decimal is a great mistake in my opinion.

The floating-point numbers should be converted to text as hexadecimal, e.g., in C99 and later, using printf with the %a or %A conversion specifiers.

Even when a human uses the textual floating-point numbers, if they are used only for making copies or comparisons, the hexadecimal numbers are more convenient and they are free of conversion errors.

How would the compiler be able to use any fast, floating point specific instructions if you parsed it as a hexadecimal string?

Not sure I understand the question. The suggestion is to use hexfloats, as they are unambiguous and straightforward to convert from and to (as opposed to anything with decimal significands): https://www.cplusplus.com/reference/ios/hexfloat/

You don't need any specific instructions. The hex encoded double contains the final binary representation of the floating point number. So no conversion is required to load it, except maybe for swapping around bytes on some architectures. Conceptually:

    double v;
    memcpy(&v, "\x00\x00\x00\x00\x00\xe4\x94\x40", sizeof(v)); // LE

On (2): implicit type promotions with unsigned and signed types can make it no-so-easy in C. FWIW, I always test with bytes with the sign bit on (on two's complement architectures, never worked with others) when I do stuff like that.

On (3): never perform equality tests on FP numbers in languages that use IEEE standard FP format.

On (4): I have worked both with ASCIIZ and counted strings, I think both have their pros and cons. But if you are writing stuff that has any contact with C/C++, it's often a bad idea to go against the stream. No, auto-adding a final 0 to your counted strings won't save the day every time.

Never perform equality tests on FP numbers period - it defeats the purpose of using floating point. This is not specific to the IEEE format.

Tell that to lua and javascript developers who only have access to FP numbers.

shrug They also only have access to IEEE FP numbers, so that applies equally much to the parent.

If my memory serves, IEEE FP is accurate for those real numbers that are also integers (at least on some range). JS users are safe as long as they stay within the integer algebra (ie they don't use division, mainly, or perhaps round up the result). Lua users have the extra option to recompile their interpreter it uses integers.

That's fair as far as it goes (I pedantically should have said general-purpose FP numbers), but, again, applies equally well (actually more so, since they were talking about IEEE FP specifically) to the parent post.

I agree generally that if you can afford the cycles to serialize and deserialize from decimal ASCII ("0.2") that you can probably afford to use a decimal numeric type in your business logic.

Personally I blame the C standard for not splitting strtod in to 1 parsing function (returning an integer and the power of ten), and 1 computation/conversion function. Without this you're basically hosed with respect to locales.

> Null-terminated strings

There is nothing inherently wrong about a null-terminated string convention.

There's no guarantee that it terminates.

I am not sure there is a universally better solution. If a function takes both string and its length (e.g. strncmp), there is no guarantee that the length is correct. Or like C++, you always represent a string by a (len,max,ptr) 3-tuple, but 1) this is not a primitive type and 2) len and max waste memory when you have millions of short strings.

If it is that bad use a variable length length code like the utf-8 transformation. For strings smaller than 127 bytes overhead is the same as NUL.

This increases code complexity. Also, for many applications, we prefer to have a max field for dynamic strings. Each type of string representation has its own pros and cons. This comes back to my original comment: there is not a universally better solution.

I'll add that variable length stuff is inefficient to parse as it breaks instruction level parallelism. Later parsing events become dependent on earlier events.

> (4) Null-terminated strings are the definition of insanity

Could you elaborate on this one? I've only used C in college and don't know why this is bad.

It's incredibly easy to do something wrong and cause bugs or security issues in your app.

Or, compare string manipulation in C with any high level language like Ruby, Python, JS and the ease of use is night and day.

As is the case in this article it often happens that having the length of a string is required for certain operations, which leads to a lot of calls to strlen. This doesn't really seem like such a big problem to begin with as one assumes that the operations that require the length also have linear complexity. However it turns out (like in the article) that this is not always the case which leads to unexpected quadratic algorithms. The pointer + length representation is only 7 bytes longer on modern architectures and does not have this problem and has a few other advantages like being able to create substrings without allocating new memory.

In fact, having the length of the string was not required. Certain ways of implementing scanf() required getting the length of the string, or otherwise scanning the whole thing.

But saner ways did not.

Besides the fact that you have to walk the string just to find the length of it...

Null-terminated strings try to save space by encoding the string length by convention. This can fail due to off-by-one errors, mistakenly allocating a fixed buffer, strings that have an unexpected null in them, and more.

Those can lead to all kinds of nasty problems like buffer overflows, which allow someone who can craft an input to write arbitrary stuff into memory.

God only knows how many security vulnerabilities and performance problems could be traced back to null terminated strings.

The decision to use 0-terminated strings in C may be the single worst decision in programming language history, at least from a security standpoint.

"in-band signaling is the root of all evil" -anonymous

Another reason is a substring of a nul-terminated string is only nul-terminated if it is a suffix. So slicing out a substring generally requires copying to a new buffer so you can append the nul byte.

I would point out two main issues:

1. Such a string cannot contain arbitrary bytes. Since it is frequently useful to work with arrays of arbitrary bytes, we have to repeat every library function twice, once for strings that can contain nuls (the mem* series of functions), and once for strings terminated by nuls (the str* series of functions). The latter are much better supported, and are easier to work with, so we frequently see programs that trip over strings with nuls in them for no particularly good reason.

2. We often want to modify strings, not just read them. Modifications often need more space than was in the original string. A nul terminated string cannot say how much room beyond the current length is allowed to be written to. By default, most library functions assume that there is enough space, however this is frequently an unsafe assumption. This again causes us to duplicate every library function twice over in order to specify the size of the buffer (the strn* series of functions). Since, again, these functions are poorly supported and harder to use, we see many many instances of programs that have buffer overflow bugs, and therefore likely RCEs, for no good reason at all.

This whole rant is a bit of a non-sequitor for a minor performance bug with a working fix.