Comment by WalterBright | Hacker News Reader

Comment by WalterBright | Hacker News Reader

Parent Full thread

The #1 undetected bug problem with C programs is buffer overflows. Experience shows it is extremely difficult to verify that arbitrary C code doesn't have buffer overflows in it. Assistance from the core language design can improve things a great deal.

D allows passing both raw pointers as parameters and pointer/length pairs. It's up to the user to choose. In practice, people have simply moved away from using raw pointers into buffers.

As for performance, in C to determine the length of a string one uses strlen(). Over and over and over again on the same string. This can be a major performance problem, even not considering the memory cache effects. When I look at speeding up C code, often the first nuggets of gold is reviewing all the explicit and implicit uses of strlen(). (Implicit uses are functions like strcat()). It's also the first place I look for bugs when reviewing C code - anytime you see a sequence of strlen, strcat, strcpy, it's often broken (typically in neglecting somewhere to account for the extra 0 byte).

All of this I agree with. In a better world 'arrays' would have added in the 1980's. The arguments about memory limitations is spurious since if you're writing good code you always pass a pointer and the length. Always no exceptions.

Yeah and all the string functions should have been marked as depreciated with C89 and fully depreciated with C99.

Yes, you can always pass a pointer and a length explicitly. And that's what the "safe" versions of e.g. string functions do. But it's still incumbent on you as the programmer to use them properly. It would still be beneficial to have a compiler mode where all that was done for you automatically and it was impossible to have a buffer overrun.

The safe string functions aren't safe because they don't return safe strings.

There is a difference between passing a length as a function argument, and actually storing string lengths alongside the strings in memory.

It's not unheard of to have millions of tiny little (< 10 character) strings, and not storing lengths alongside them can shave off a sizeable portion of space requirements.

Anytime a program has a special case like that, it makes sense to craft an optimal data structure for it.

Also, consider that the terminating 0 byte isn't just one byte. There's also the alignment of what malloc returns, which may be 4 or 8 or even 16 bytes.

malloc also has internal bookkeeping overhead, typically 16-32 bytes per allocation.

Which is why one should never allocate a single (short) string from a generic allocator. Instead, one allocates a big chunk upfront (e.g. 4K bytes or more), and breaks from that, using a simple index that points to the first unused bytes.

In this way, the overhead of allocating a string is truly only the terminating zero byte - no alignment constraints. This scheme is easy to implement as long as strings don't need to be freed individually.

I must be a really really good programmer, since I rarely see the need to use strlen().

For one, strings are just chunks of memory like other arrays. So for almost any string that is not a literal in the source code, you just store offset/length as needed, like for any other array. I have sizeable projects (on the order of 10K lines) that have maybe 0 or 1 instances of strlen() in the code.

Very often though, strings are bounded and pretty short (since they are meant for human consumption), and in these cases, using strlen() when looking up a string is often sensible, since persisting the length might require more memory than the string itself.

Another case is when you're scanning the string from left to right anyway, so you just "stream" through it until you find the terminating 0. That's how printf() and friends work (they take a formatting string), and arguably this scheme works just fine.

Btw, the length of a string literal is (sizeof "Hello" - 1). You can also initialize char arrays using string literals and have the size available:

    static const char name[] = "Foobar";
    static const int nameLen = sizeof name - 1;

Look how many C library functions implicitly call strlen() internally.

Also, look at functions like sscanf(). It can be orders of magnitude slower than fscanf() because every invocation calls strlen while fscanf incrementally reads from the current file pointer. I don't know why sscanf doesn't also work incrementally but the implementations I've tested don't do that.

The main point here being: if strings had a size_t size plus data, that would change an O(n) scan to an O(1) length lookup, and that would have huge performance gains throughout the C library, not to mention your own code as well.

I checked musl libc and the way the implementation implements sscanf is by calling vsscanf with a custom FILE stream. And that file stream is implemented using __string_read(), which does indeed call strnlen().

I figure it would be possible for musl to implement that stream using a function that scans for NUL and copies at the same time, but maybe that's not an improvement in the end.

It would be much simpler of course if sscanf() would take the length of the input string as an additional argument. But actually, I don't really care.

Because, does this even matter? Using sscanf() is far from ideal anyway. The stdio functions are not what you use if you're going for performance. Their conversions are probably not the fastest (being quite featureful), and they are even locale dependent which is a huge mess!

Heck, when we're going for performance to a degree where a strlen() matters (bear in mind that we have to read the input at least once anyway, so the waste is definitely bounded) we should certainly not be parsing text at all. That is much more wasteful in comparison.

Much if not most of libc is there to provide you a portable base to (comparatively) quickly get your project up and running, and to simply to keep old software going, but it's certainly not to help you achieve performance.

http://git.musl-libc.org/cgit/musl/commit/?id=18efeb320b763e...

If you ask printf to right justify a string, it'll do a strlen.

I don't agree that in the days of valgrind, asan etc. the #1 issue is buffer overflows.

#1 and #2 are integer overflows and aliasing mistakes.

valgrind is a marvelous tool, but it only detects actual buffer overflows, not vulnerability to buffer overflows.

Reply on news.ycombinator.com