Hacker News Reader

Comment by adwn | Hacker News Reader

> There is little of needless code in standard library. And what there is, believe me, over the decades was really polished.

Wrong. Counter-example: locales [1]

Also: String handling, which is responsible for so, so, so many security vulnerabilities. The underlying cause is zero-terminated strings (instead of using start/end pairs or start/length pairs). You can't even tokenize a string without either copying or modifying it!

Also also: just a single, apologetic mention of undefined behavior. Responsible – in cooperation with over-zealous compiler writers – for so many more bugs not already caused by improper string handling.

[1] https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f02...

View on HN

I love C but I think most of the C standard library is useless and outdated rubbish (pretty much everything above the mem...() and math functions). The entire IO and memory-management areas should have been either thrown out of the standard library, or updated 20 years ago to keep up with low-level operating system features.

But one of the best features of C is that you can mostly ignore the standard library and still enjoy "C the language", e.g. nobody ever choose C for its standard library ;)

(also re UB etc...: use the mighty trio UBSAN, ASAN and TSAN!)

What do you use then if you don't use the standard library?

What is the standard library for the C which ends all the other standards libs?

Your own.

For IO, memory management and similar "low-level services" it's often fine to call the underlying operating system functions directly and with this also make use of more powerful features than the standard library can provide, such as async IO, virtual memory etc (or wrap those OS calls into your own cross-platform wrapper functions). There's also plenty of libraries which do this for you, like libuv (but integrating such big libs is often not trivial).

Same for string handling/processing, there are specialized libraries out there which do a specific job better than the rather generic standard library functions (this is also true for the C++ stdlib), one just has to find those libraries.

PS: I don't think that a "batteries included" standard library (like python has) would even make sense for C, such a library would need to be opinionated by definition, better to keep (too many) opinions out of the standard, as this road just leads to another C++ ;)

C++ is IMO defined by a complete lack of opinion. They just standardized every opinion. For everything. C++ would be a much better language if it showed some opinion, even if I disagreed with it.

That's honestly how I also feel about C++ in when I look at how often they are adding new things.

> Same for string handling/processing, there are specialized libraries out there which do a specific job better

I actually can't think of a lot of these that are super common across projects.

ICU is probably the most notable one, with its particular focus on unicode correctness.

A lot of large projects end up writing their own string library of sorts.

There's bstr, I've come across that a few times.

In my long C career, I invented string library after string library. They all turned out to be garbage. The common cause of their failures is their inability to interact with any other C code.

Agreed, but that's an issue for all "interface types" on API boundaries (another similar issue is vector and matrix types in math libraries). IMHO a C string library must still be able to efficiently consume and produce vanilla zero-terminated "const char*" string data (and at a stable address, unlike for instance std::string) even if they mostly work with {pointer,length} ranges internally.

C strings are so ubiquitous that the ABI impedance mismatch is just too annoying.

You can use something like GitHub or Conan to find specific useful C libraries.

There isn't a universally recognized "standard library replacement" (and people argue about what parts of the standard library they don't like), but here:

https://stackoverflow.com/questions/486383/safer-alternative...

are a few options:

* Glib (the basis for Gtk): https://developer.gnome.org/glib/stable/glib.html

* The Apache Portable Runtime (APR): http://apr.apache.org/

Caveat: I haven't used them.

djb's libraries (answer to first question)

> But one of the best features of C is that you can mostly ignore the standard library and still enjoy "C the language", e.g. nobody ever choose C for its standard library ;)

C is one of the hardest languages to do this with given its anaemic dependency management.

> (also re UB etc...: use the mighty trio UBSAN, ASAN and TSAN!)

All of which will miss some cases, even in combination.

There's a lot of subtly broken stuff in C stdlib. For example, time.h calls that may touch the TZ env var aren't thread-safe, because getenv is unsafe (it gives a pointer to its internal globally mutable data structure without locks).

Because C is so bare-bones, it usually leans on POSIX as its extended standard library, and that is also full of old cruft.

One thing that is not-so-subtly broken is re-entrancy (e.g. in functions like `strtok()`; fixed later with `strtok_s()` in C11:

https://en.cppreference.com/w/c/string/byte/strtok

About getenv: This sort-of fixed with `getenv_s()` in C11

https://en.cppreference.com/w/c/program/getenv

Re: getenv_s: Annex K is (still) optional. Compliant C11+ implementations may not have getenv_s.

This is an understatement. The only relevant platform that supports Annex K is MSVC, and this is unlikely to change any time soon.

And even MSVC does not actually implement the API specified in Annex K. Annex K was inspired by MSVC's design, but they diverged:

> Microsoft Visual Studio implements an early version of the APIs. However, the implementation is incomplete and conforms neither to C11 nor to the original TR 24731-1. For example, it doesn't provide the set_constraint_handler_s function but instead defines a _invalid_parameter_handler _set_invalid_parameter_handler(_invalid_parameter_handler) function with similar behavior but a slightly different and incompatible signature. It also doesn't define the abort_handler_s and ignore_handler_s functions, the memset_s function (which isn't part of the TR), or the RSIZE_MAX macro.The Microsoft implementation also doesn't treat overlapping source and destination sequences as runtime-constraint violations and instead has undefined behavior in such cases.

[0]: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1967.htm

strtok_r() is everywhere and easy to retrofit. It's also easy to implement using strspn() and strcspn() if you don't have it.

This makes sense when you consider the history of C. Multi-threading was a relatively late add on, and outside of the standard library.

When I learned C (early 90's), multi-threading wasn't even something you considered. It was either multiple processes or an event loop with select/poll.

Locale handling and most of the str* function in libc indeed are quite bad. The only sane function for "C strings" is snprintf.

Ones like strspn, strcspn, strpbrk, strchr are also sane.

True. With the snprintf comment I was mostly pointing out str* functions that are designed to either modify or concatenate strings.

Except when you consider what a char is. Those don't handle multi-byte sequences.

Yes they do. What they don't do is search for multi-byte characters. You would not use them for that; you use them to look for ASCII delimiters. You can tokenize a UTF-8 string on ASCII delimiters using functions that are oblivious to UTF-8, and this is in fact a preferred technique in many programs.

I think I was going for more like: strpbrk and friends can't use a multi-byte sequence as a delimiter. And if a delimiter appears as part of a multi byte sequence, you may see strange results.

In a localized world searching for delimiters also starts to make less sense. Eg. I have been told it doesn't make sense to break on whitespace for Chinese text.

To be clear, I am not saying these functions are bad or evil or to blame for their limitations (a bunch of the problem space wasn't invented yet when they were introduced), just noting they have limits. A bunch of more recent languages and libraries have the same or similar issues, too.

Use wcspbrk.

UTF-8 continuation characters are limited to the range \200 through \300 so there's basically zero chance that if you choose something like comma as your delimiter that it's going to tokenize the middle of a multibyte sequence.

Also take into consideration that, under the hood, functions like strpbrk() are typically accelerated by CPU instructions such as PCMPISTRI which doesn't support UTF-8 natively but it does support UCS-2.

> so there's basically zero chance that if you choose something like comma as your delimiter that it's going to tokenize the middle of a multibyte sequence.

Not just "basically;" there is no possible collision between ASCII characters and any valid multibyte encoding. This can be seen somewhat visually in this table[1] and is an intentional aspect of the UTF-8 design.

[1]: https://en.wikipedia.org/wiki/UTF-8#Encoding

How about with joiners and combining characters? Eg. If you encode é as U+0065, U+0301 (\x65\xcc\x81), then search for 'e' and act on the result somehow, you fail to consider the whole glyph.

Sure. You're talking about glyphs that are composed of multiple unicode codepoints; my earlier comment is true of single codepoints only. The comment I was responding to is also talking only about single codepoints (wcspbrk cannot represent delimiters longer than a single codepoint).

On joiners / combining characters: I'd encourage using composed normalization (NFC) rather than decomposed normalization (NFD).

Just curiosity: are there any glyphs that lack a single codepoint representation, where one of the joined codepoints is an ASCII character? (That only helps after normalization, of course.)

Yes. ASCII uses \b as the combining character mark which is a convention that's always been widely supported by typesetting programs such as less and nroff. For example, A\b_ is A̲, and you can do the same thing with apostrophe and tilde for accent marks. There's also UNICODE emojis where two codepoints in sequence get joined together as a single glyph. Never underestimate the creative ways text can be used, or that standards just codify a long history of practices.

Er, I was asking about unicode joining, not this roff \b thing. Sorry for the confusion. I'm aware that multiple-codepoint unicode glyphs exist; I'm asking if any of those involve a codepoint in the ASCII (1-127) range which cannot be normalized to a single codepoint (e.g., e + ' normalizes to a single codepoint é).

Of course. Take for example mͫ (m+m) there's no way to represent that as a single codepoint. Combining marks can also be overlaid multiple times, e.g. m͚ͫ (m+m+∞) so the number of glyphs you can create is limitless. There's only a tiny number of the combinations that are possible which have a tinier normalized form. The new UNICODE combining marks work by almost exactly the same principles as the \b ASCII combining mark. That's why I mentioned it earlier.

Thanks!

In what data format or programming language is 'e' a delimiter? One situation is floating-point constants, where 'e' is a delimiter indicating the exponent. However, if an é occurs in the middle of such a constant, whether as a single code point or a combined character, that is an error. The 'e' must be followed by an optional sign and one or more decimal digits.

The ISO C library string handling stuff is for systems programming, not for scanners and parsers for natural written language.

strstr can find a multi-byte character in a multi-byte sequence. There is no ISO C function to find the first occurrence of any one of a bag of multi-byte characters, though it's not difficult to write one.

> I have been told it doesn't make sense to break on whitespace for Chinese text.

It could make sense to break on whitespace in some programming language or data format that allows Chinese (and other) identifiers.

A command interpreter that allows Chinese arguments (such as file names) wants to break on spaces, as usual.

> if a delimiter appears as part of a multi byte sequence, you may see strange results.

UTF-8 was designed by a dyed-in-the-wool C-and-Unix engineer, who ensured that such a thing can't happen. No character in the 0x00-0x7F range can occur in a multi-byte character.

The byte which starts a UTF-8 character cannot occur anywhere other than at the start, which is why we can use strstr to look for it.

It's been a long time since I've used C, but I recall strncpy being safe. I assume from context that I'm wrong, though.

Every time I review code and see strncpy, I look closer because it's always used incorrectly. It's always about the terminating 0. Is the 0 there or not? Is the 0 part of n or not? Does the destination have to be n+1 in size for the 0?

I quit using it myself because I could never remember just what the exact protocol was for 0.

Yeah used naively strncpy leaves you an unterminated string. Also like all of them it's up to the caller to predetermine if they will fail if called. So instead of having the checks in one place inside the string function. You have them scattered all over the code if at all.

The irony of strncpy is it's supposed to eliminate memory corruption errors, but instead is a prolific source of them.

I think I read somewhere the provenience of strncpy was to copy strings into a fixed length field which is why it has the deranged behavior of not terminating the string. Think file systems where the max file name is 8 characters. Or compilers that truncated variable names at 31.

Strncpy doesn't add a \0 if it truncates the string, so the user has to remember to do that.

strncpy is one of the trap functions. It's better avoided. Write your own or use snprintf.

That [1] is a top-quality rant, but I fear for the author's blood-pressure

Exhibit B: yes, the command that repeatedly prints `y` to stdout. Exhibit C: all handling of time, with time_t, struct timespec, timeval etc. We should just have a 64 bit uint representing nanoseconds from epoch, which would give us 585 years and get rid of much nonsense.

[1] https://github.com/coreutils/coreutils/blob/master/src/yes.c

Time should be a signed int with enough bits to go backward from the epoch at t=0 to the big bang at t=negative whatever.

In what resolution?

What do people that need higher resolution use? And the people that don't care about that amount, do they pay the performance penalty?

Every physical or humane value is hard to some extent.

> enough bits to go backward from the epoch at t=0 to the big bang at t=negative whatever.

For reference, the universe is about 13.787 billion years old [0]. That's about 13.787 * 10^9 * 365 * 24 * 3600 = 4.348 * 10^17 seconds, which (I think?) is a 59-bit number [1]. 10ths of a second will require 62 bits, which is right about at the edge of what a 64-bit signed integer will allow.

If you want milliseconds, you'll need at least 69 bits. For nanoseconds, you'll need at least 89 bits.

So you'll either need an integer type that's wider than what's natively supported in most hardware (thus potentially sacrificing performance), or you'll have to sacrifice precision.

[0]: https://en.wikipedia.org/wiki/Age_of_the_universe

[1]: https://www.wolframalpha.com/input/?i=13.787+*+10%5E9+*+365+...

Just go for it. In the larger scheme of things, it's not that long until 128-bit CPUs.

I'm mostly kidding. But if you're thinking of representing nanoseconds since the big bang, the wait for 128-bit CPUs is not very long...

GNU has extensions to support 128bit number types [0] already, for systems that can handle it.

[0] https://gcc.gnu.org/onlinedocs/gcc/_005f_005fint128.html

struct timespec already exists and stores time as two 64 bit integers. Should last a little while.

Ah, that's something I wasn't aware of. Yeah, that scheme could work if you didn't need a single integer to represent time.

So, the problem is a 64 bit int only gives you ~500 years around the epoch at nanosecond resolution. That's sufficient for timestamping, but not great for a general purpose time library where even ordinary civil calculations might exceed that range, particularly in intermediate arithmetic.

Since larger than 64 bit ints are a disaster for portability, the reasonable solution is to go with a 64 bit signed seconds, 32 bit nano offset field. A lot of language std libs have adopted something along these lines.

DJB was advocating for everything to be in a format like this, referenced to TAI (UTC without leap seconds, basically). Sadly that didn't get any traction.

I remember in my University years i was astounded to find out that none of the programming languages we were taught could represent the Jurrasic period, or the day the earth was formed, as a date.

I was horrified even more when i learned that future leap seconds are undefined, and we literally can't tell what is the time on the clock lik3w a million seconds from now.

This only really really works if you do TAI. Down with leap seconds.

...? `yes` is not part of the C standard library.

I think they meant to draw attention to the implementation of `yes` to showcase crustiness the C stdlib imposes on even a trivial program.

But of course GNU is kind of notorious in this regard. Compare their `yes` to OpenBSD's. It's night and day.

https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c

It's not that GNU is incompetent or overly bureaucratic or anything. It's that they really didn't want to infringe on Unix, and they presumed that Unix's yes would have been implemented in the most obvious, dead-simple way. And, as a plus, GNU yes can stream y's at like 10 GB/s, if you need that.

In my firmware I keep time as a 64 bit signed number that counts 32.768 khz ticks since the unit powered up. I think it'll roll over in 8 million years or something like that.

If you want to know what's broken? Most real time clock modules. They almost all want to store time as HH:MM:SS MM:DD:YY and sometimes 1/256 of a second but sometimes not.

the over-zealous compiler writers is getting a big problem. I remember a while ago there was a proposal of a friendly c standard.

Indeed; by John Regehr: https://blog.regehr.org/archives/1180

I don't think it has really been proposed anywhere though, unfortunately.

I have been surmising there might be copyright issue of the actual standard?

> The underlying cause is zero-terminated strings (instead of using start/end pairs or start/length pairs).

What type (and how big) should length be?

Probably something like how msgpack (or even variable length unicode encoding) works, where the low 127 bytes is a single uint8 char in place, and then the next 121 signals that a string of length 1-121 bytes follows, and then after that are a handful of sentinel values for "length follows and it's 1/2/4/8 bytes" plus maybe some special cases like zero-length string.

I can understand that this might have been too much implementation complexity/risk to contemplate 40 years ago, but this kind of pattern is very well established at this point, especially in scripting languages with loose typing.

> What type (and how big) should length be?

size_t.

> Also: String handling, which is responsible for so, so, so many security vulnerabilities.

That's not really a counter-example. Sure the string handling was an unfortunately choice but it is the standard. The standard library must implement it as defined. Being standard-compliant is not lack of polish.

Depends on what one means by "polished", I suppose. You can have a polished implementation of a not-so-polished specification, and vice versa.

> You can't even tokenize a string without either copying or modifying

rust-lang saves the day, yet again https://www.brandonsmith.ninja/blog/favorite-rust-function

One word: strtok

It takes more code to use it correctly than to hand-code what you think it is supposed to be doing for you. strtok is Cursed.

strlcpy is similar. If you don't write that much more code, you are not using it correctly, and it is not giving you the value that is the reason you thought was why you were using it.

A single counter-example doesn't make it all wrong.

Perhaps a committee was involved.

Reply on news.ycombinator.com